Real-time Rigid Body Visual Tracking for Nanomanipulation and Characterization
As researchers continue to investigate the nano domain, robotic tools are increasingly called upon to perform more complex manipulation tasks. These operations are primarily open-loop, due to the limited number of feedback options with the required precision. By closing the loop for positioning control with the end effector, a variety of strategies such as bilateral telemanipulation or automated manipulations become possible. One source of this feedback is the image generated by an electron microscope. Real-time visual feedback from electron microscopes are typically noisy and pose significant challenges to an image processing system. This work proposes using rigid model based algorithms for object tracking in a scanning electron microscope. The use of domain specific knowledge by the introduction of two- or three-dimensional object models can be used to provide extra information to the tracking process and increase the system precision.
When manipulating objects at the nano scale sensor feedback becomes a challenging task. Typically, piezoelectric actuators are used in nanomanipulation tasks due to their ability to make small displacements and the absence of backlash. One drawback of these actuators, when compared to DC motors, is due to the lack of gearing, the encoder feedback is unable to capitalize on a multiplication of resolution. Instead, they are limited to the precision of displacement sensing technology. In the case of stick-slip piezoelectric actuators (SSA), the devices often rely on optical or resistive strategies, which have an accuracy on the order of ones or tens of microns. This lack of precise positioning information makes task space control of kinematically complex robots difficult without a further source of feedback.
Manipulations are often performed in conjunction with devices such as optical microscopes, scanning electron microscopes (SEM), or transmission electron microscopes (TEM) which all provide sensory feedback in the form of an monocular image. The SEM is often favored for nanomanipulation due to its high resolution and high depth of field. These image based sensors can be leveraged to increase the system's positioning resolution.
One major challenge of performing visual servoing tasks inside a SEM is balancing the needs of image quality with real-time imaging. The sequential scanning used to create a SEM image necessitates lower framerates than those available with optical cameras. Also SEM imaging quality is highly dependent on the scale and material properties of the area being viewed. Many times these parameters cannot be changed because they are directly related to the task being performed. Thus, methodologies must be developed to work with the limited framerates and imaging restrictions of the SEM.
In particular, we are investigating CAD model based methods to help provide more precise motion feedback. In the micro and nano domains, many of the parts of interest are previously modeled for manufacturing processes. The task then becomes iteratively finding the modeled points in the image and determining a pose that corresponds. Due to the limited resolvability along the optical axis of the system, the algorithm's precision is inherently limited. Thus, once the object is tracked within the limits of the system, strategies must be developed for visual servoing a manipulator using an monoscopic, orthographic view of the scene.
Videos
This video demonstrates the applicability of selective scanning to Scanning Electron Microscopy (SEM) imaging. The video first demonstrates a full frame scan of the motion in which the microgripper moves large distances between frames due to the slow imaging rate. By tracking the gripper and selectively scanning regions of the image, we are able to icrease the frame rate from ~4 Hz to ~15 Hz and thus improve the tracking and subsequent manipulation feedback.
