9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

Changyi Lin1, 2,  Han Zhang2, 1,  Jikai Xu3, 1,  Lei Wu3, 1,  Huazhe Xu2, 4, 1
1Shanghai Qi Zhi Institute,  2Tsinghua University, 
3Huazhong University of Science and Technology,  4Shanghai Artificial Intelligence Laboratory
ArXiv Hardware & Code Dataset Tutorial (en) Tutorial (cn) Summary

Key Features!

Accurate 3D shape reconstruction with only white light!
Generalizable 6D force estimation without markers!
Compact, Robust, Low-Cost, Easy-to-Fabricate!


Motivation for 3D Shape Reconstruction

We leveraged the reflective properties of silicone. Black silicone absorbs light, while translucent silicone reflects it. By stacking two different types of silicone gel and placing an LED and a camera below, we observed a uniform reflection from these layers. When an object contacts the gel surface, the translucent gel becomes thinner under pressure, reducing light reflection in these areas, which appear darker in the image. It means that the deeper you press, the darker you will see. We have evaluated this idea in the DTact paper.

Design of 9DTact

9DTact is remarkably compact, with the size of only 32.5mmx25.5mmx25.5mm.
It features a relatively large sensing area, resulting in a small dimension to sensing area (D/A) ratio.
It is adaptable to be installed in a wide range of robotic platforms, from grippers to dexterous hands.

Other Improvements

1. Optimize the acrylic board for more stable tactile images;
2. Design an isolation ring for more uniform tactile images;
3. Refine the black gel layer process to make 9DTact more robust to environmental light source.

Calibration for 3D Shape Reconstruction

We develop a virtual marker-based image rectification method to simultaneous calibrate these parameters:
1. The lens distortion;
2. The pixel position on the tactile image that corresponds to the sensor surface's central position;
3. The physical length on the sensor surface corresponding to one pixel of the tactile image.
As for pixel-depth calibration, 9DTact only requires one contact image by pressing a metal ball.

Results of 3D Shape Reconstruction

MAE: 0.0462mm, Std: 0.0304mm.


Motivation for 6D Force Estimation

As a hyper-elastic material, the pressed gel tends to flow outward to its neighboring regions, resulting in an increase in thickness in the surrounding areas. Similarly, when the contact object applies shear force or twist force, the surrounding gel will flow to accumulate along the moving direction. Furthermore, according to the principle that thinner contact areas result in darker pixels, the thicker surrounding regions induce brighter pixels.

Gel FLow Visualization

We visualize the areas becoming darker with red color and brighter areas with green color, the green pixels turn to be much brighter in the object's moving direction. With the darker areas extracting the concave deformation information, and the brighter areas extracting both the convex deformation and shear deformation information, deformation in all directions can be captured.



Dense Deformation Representation

1. Darker Channel: contact geometry;
2. Brighter Channel: the dragging and twisting motions of the contact geometry;
3. Reference Channel: a sensor's reference state.

Data Collection for Force Estimation

We collect over 100K image-force pairs on 175 complex objects, with autonomous image selection.

Results for 6D Force Estimation

Standard splitting: 10000 pairs of data are randomly selected from all data as test set;

Object-based splitting: 18 of the 175 objects are randomly selected as test objects.

Testing examples.

Demos on Unseen Objects


Fully Open-Source

We have fully open-sourced both the hardware and software of 9DTact on Github.
Follow the video instructions to build your own 9DTact tactile sensor!

Tutorial in English

Tutorial in Chinese

Abstract

The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution.

In this work, we present 9DTact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and DIY-friendly, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100,000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects.

To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact as well as present a 1-hour video tutorial.

BibTeX

@article{lin20239dtact,
  title={9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation},
  author={Lin, Changyi and Zhang, Han and Xu, Jikai and Wu, Lei and Xu, Huazhe},
  journal={arXiv preprint arXiv:2308.14277},
  year={2023}
}