Dextrous Code Generation:
Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Tasks

Abstract

Large Language Models (LLMs) have been successful at generating robot policy code, but so far these results have been limited to high-level tasks that do not require accurate movement. It is an open question how well such approaches work for tasks that require reasoning over contacts forces and working within tight success tolerances. We find that, with the right action space, LLMs are capable of successfully generating policies for a variety of contact-rich and high-precision manipulation tasks. Specifically, we reparameterize the action space to include compliance with constraints on the interaction forces and stiffnesses involved in reaching a target pose. We validate this approach on subtasks derived from the Functional Manipulation Benchmark (FMB) and NIST Task Board Benchmarks, where policy generation with an LLM in this action space improves success rates over generation in non-compliant action spaces by greater than 3x and 4x, respectively.

Experiments

Here is a showcase of the experiments we ran.

Functional Manipulation Benchmark Prompt
Circle

"Explain how to insert a circle-shaped peg into a circle-shaped hole with a grid pattern search. Then, generate the code to insert a circle-shaped peg."

Generated code

Star

"Explain how to insert a star-shaped peg into a star-shaped hole is different than a circle-shaped peg with a grid pattern search. Then, generate the code to insert a star-shaped peg."

Generated code

Half-Pipe

"Explain how to insert a half-pipe-shaped peg into a half-pipe-shaped hole is different than a circle-shaped peg with a grid pattern search. Then, generate the code to insert a half-pipe-shaped peg."

Generated code

Assembly Prompt
Cable routing

Currently the robot is holding the cable at pose(1), but the cable is above a tunnel with a small opening at the top. Describe a strategy to search for an opening and insert the cable using force information and then write a program using the APIs above to route the cable into the tunnel.

Generated code

Cable un-routing

"Currently the robot is holding the cable at pose(1), but the cable is stuck in a tunnel with a small opening at the top. Describe a strategy for unrouting the cable using force information to search for the opening and then write a program using the APIs above to unroute the cable and lift it out of the tunnel."

Generated code

Connector insertion

"Now assume the robot is holding a waterproof connector above a port. Use the above methods to perform a grid pattern search insertion. Then rewrite it to remove any ifs or while loops."

Generated code

Cable unravel

Prompt

Generated code