Leading commercial robot-assisted minimally invasive surgery (RMIS) platforms such as the da Vinci Surgical System do not provide force feedback to the surgeon. Instead, surgeons rely on visual force estimation to judge the amount of force they apply to tissue. This is a difficult skill that takes a long time to master. Experienced surgeons often report being able to 'feel' tissue manipulation forces in their hands. Motivated by the need to accelerate the learning of visual force estimation in RMIS, we investigate the effects of training with force feedback on visual force estimation during teleoperated soft environment manipulation.

Z. Chua, A. M. Jarc, S. Wren, I. Nisky, and A. M. Okamura, “Task Dynamics of Prior Training Influence Visual Force Estimation Ability During Teleoperation,” IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 4, pp. 586–597, 2020.

 

Vision-based Force Estimation and Haptic Feedback Using Neural Networks

Forcep tip force sensing in robot-assisted minimally invasive surgery is challenging due to strict requirements for miniaturization, biocompatibility, and sterilizability. Indirect force estimation is a promising method to measure forces while circumventing these constraints. Much like how humans can estimate forces visually, neural networks can attempt to do something similar. However, there are concerns as to the generalizability of these methods as well as the relative importance of visual information compared to robot kinematic state information as inputs. We characterize the performance of vision-based neural networks with these considerations in mind and quantify the quality of the closed-loop haptic feedback they can provide to the operator.

Z.Chua and A. M. Okamura, “Characterization of Real-time Haptic Feedback from Multimodal Neural Network-based Force Estimates during Teleoperation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022.

Z. Chua, A. M. Jarc, and A. M. Okamura, “Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State,” in IEEE International Conference on Robotics and Automation, 2021, pp. 12335–12341.

 

Force Sensors for Minimally Invasive Surgery

Research into haptic feedback, data-driven indirect force sensing approaches, and surgical skill evaluation require accurate knowledge of tool-tissue interaction forces. However, such measurement tools are not available commercially, and are difficult to realize without specialized knowledge. Our group develops sensor designs that strike a balance between ease-of-manufacture, cost, size and accuracy, to help researchers achieve their vision, thereby helping to advance our field. Our force sensors use an array of miniature commercially available load cells to sense forces applied to the forcep jaws. The interface to different forcep jaws are compatible with different jaw geometries making the system customizable to different surgical applications. The range of the for sensor is 5N in axial compression, and tension and 3N in the positive and negative lateral directions.

Z. Chua and A. M. Okamura, “A Modular 3-Degrees-of-Freedom Force Sensor for Robot-Assisted Minimally Invasive Surgery Research,” Sensors, vol. 23, no. 11, p. 5230, 2023.