By Adam Zewe
A human clearing junk out of an attic can often guess the contents of a box simply by picking it up and shaking it, without the need to see what’s inside. Researchers from MIT, Amazon Robotics, and the University of British Columbia have taught robots to do something similar.
They developed a technique that enables robots to use only internal sensors to learn about an object’s weight, softness, or contents by picking it up and gently shaking it. Their method does not require external measurement tools or cameras, the robot can accurately guess parameters like an object’s mass in seconds.
This low-cost technique could be useful in applications where cameras might be less effective, such as sorting objects in a dark basement or clearing rubble inside a partially collapsed building after an earthquake.
What is key to this approach?
Key to their approach is a simulation process that incorporates models of the robot and the object to rapidly identify characteristics of that object as the robot interacts with it.
The researchers’ technique is as good at guessing an object’s mass as are complex and expensive methods incorporating computer vision. In addition, their data-efficient approach is robust enough to handle a variety of unseen scenarios.
“This idea is general, and I believe we are just scratching the surface of what a robot can learn in this way. My dream would be to have robots go out into the world, touch things and move things in their environments, and figure out the properties of everything they interact with on their own,” says Peter Yichen Chen, an MIT postdoc and lead author of a paper on this technique.
His co-authors include fellow MIT postdoc Chao Liu; Pingchuan Ma PhD ’25; Jack Eastman MEng ’24; Dylan Randle and Yuri Ivanov of Amazon Robotics; MIT professors of electrical engineering and computer science Daniela Rus, who leads MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL); and Wojciech Matusik, who leads the Computational Design and Fabrication Group within CSAIL. The research will be presented at the International Conference on Robotics and Automation.
Sensing signals
The researchers’ method leverages proprioception, which is a human or a robot’s ability to sense its movement or position in space.
For instance, a human who lifts a dumbbell at the gym can sense the weight of that dumbbell in their wrist and bicep, even though they are holding it in their hand. In the same way, a robot can “feel” the heaviness of an object through the multiple joints in its arm.
“A human doesn’t have super-accurate measurements of the joint angles in our fingers or the precise amount of torque we are applying to an object, but a robot does. We take advantage of these abilities,” Liu says.
As the robot lifts an object, the researchers’ system gathers signals from joint encoders, sensors that detect rotational positions and speed of its joints during movement.
“Most robots have joint encoders within the motors that drive their movable parts,” states Liu. “This makes their technique more cost-effective than some approaches because it doesn’t need extra components like tactile sensors or vision-tracking systems.”
How do you estimate an object’s properties?
To estimate an object’s properties during robot-object interactions their system relies on two models: one that simulates the robot and its motion and one that simulates the dynamics of the object.
“Having an accurate digital twin of the real-world is really important for the success of our method,” Chen adds.
Their algorithm “watches” the robot and object move during a physical interaction and uses joint encoder data to work backward and identify the properties of the object.
For instance, a heavier object will move slower than a light one if the robot applies the same amount of force.
Differentiable simulations
They utilize a technique called differentiable simulation, which allows the algorithm to predict how small changes in an object’s properties, like mass or softness, impact the robot’s ending joint position. The researchers built their simulations using NVIDIA’s Warp library, an open-source developer tool that supports differentiable simulations.
Once the differentiable simulation matches up with the robot’s real movements, the system has identified the correct property. The algorithm can do this in a matter of seconds and only needs to see one real-world trajectory of the robot in motion to perform the calculations.
“Technically, as long as you know the model of the object and how the robot can apply force to that object, you should be able to figure out the parameter you want to identify,” Liu says.
The researchers used their method to learn the mass and softness of an object, but their technique could also determine properties like moment of inertia or the viscosity of a fluid inside a container.
Plus, because their algorithm does not need an extensive dataset for training like some methods that rely on computer vision or external sensors, it would not be as susceptible to failure when faced with unseen environments or new objects.
In the future, the researchers want to try combining their method with computer vision to create a multimodal sensing technique that is even more powerful.
“This work is not trying to replace computer vision. Both methods have their pros and cons. But here we have shown that without a camera we can already figure out some of these properties,” Chen says.
They also want to explore applications with more complicated robotic systems, like soft robots, and more complex objects, including sloshing liquids or granular media like sand.
In the long run, they hope to apply this technique to improve robot learning, enabling future robots to quickly develop new manipulation skills and adapt to changes in their environments.
This work is funded, in part, by Amazon and the GIST-CSAIL Research Program.
Read next: Inspiring the Next Generation – Build in Digital
Are you a building professional? Sign up for a FREE MEMBERSHIP to upload news stories, post job vacancies, and connect with colleagues on our secure social feed.
