Section 43.3: In-hand manipulation and reorientation

"The hard part of dexterity is not holding on, it is changing contact without losing meaning."

A Dexterous Manipulation Bench

Pick up a pen, then rotate it into writing position using only the fingers of that one hand: you just executed a sequence of rolling, pivoting, and finger-gaiting contact transitions that no robot can yet perform reliably outside a lab. That silent regripping is what in-hand manipulation is, and it forces the robot to reason over object pose and hand configuration together. A hand that cannot reorient what it holds is not dexterous: it is a clamp with ambitions, as Figure 43.3A illustrates.

Finger gaiting, where fingers release and re-place contact points one at a time so the remaining fingers keep the object stable, matters in embodied AI because it is the only contact mode that can reposition an object when the required rotation exceeds what rolling alone can deliver within the hand's joint limits. Without it, a robot is permanently constrained by the initial grasp geometry: any object pose outside the reachable roll arc requires either a full regrasp or whole-arm motion, both of which are slower and increase drop risk on a physical platform.

How finger gaiting moves the object

Mechanically, finger gaiting briefly lifts one finger off the object. That finger swings to a new contact point while the remaining fingers maintain force closure, then re-establishes contact at the new location. The static fingers support the object throughout. It is the robotic equivalent of repositioning your grip on a wet bar of soap one finger at a time: the moment any finger loses contact, every remaining finger must silently absorb the load shift or the object is already falling. Each gait step shifts the reachable orientation arc, so the planner can string together multiple steps toward orientations unreachable in a single roll.

It extends grasping from one-shot acquisition to reorientation, tool positioning, and dexterous correction, where a single grasp score stops being enough. Figure 43.3.1 casts this as a closed loop: observe pose and contacts, plan a contact sequence, act by rolling or regrasping, verify the orientation residual, and repeat until the target is reached.

Theory

This theory section builds, piece by piece, the contact-sequence planning framework promised above: first the state and action variables the planner reasons over, then a concrete system that shows why reachability (not raw force) is the bottleneck, and finally the reachability equation the algorithm callout below actually searches over.

The state includes object orientation, fingertip contacts, joint configuration, and latent slip state. The action is often a combination of finger motion and controlled object motion induced by rolling or pivoting contacts.

Consider a specific case: OpenAI's Dactyl system (Andrychowicz et al., 2019) trained a Shadow Dexterous Hand, a 24-degree-of-freedom anthropomorphic robot hand commonly used as a research platform for dexterous manipulation, to reorient a cube to arbitrary target orientations using reinforcement learning with domain randomization. In the reported results, the system achieved average rotations of roughly 50 face rotations before dropping the object, using no tactile sensors, relying entirely on vision and wrist proprioception. This suggests that the core bottleneck is typically not hardware richness but the quality of the contact-sequence policy and the robustness of pose estimation inside the hand.

This is why reorientation cannot be reduced to a single grasp quality number. The planner must ask whether the current contact set admits a path to the target pose through reachable intermediate contacts. In the same reported results, a naive open-loop finger motion policy dropped the cube after roughly 3 rotations on average, while a policy that explicitly reasoned over contact-reachable intermediate poses reached 50 or more before any drop, on the same hardware.

To turn that reachability intuition into something a planner can search over, we state the target orientation as a product of incremental rotations constrained to reachable contact sets. The equation below spells out that constraint: the target object orientation \(R_{o,T}\) is reached by composing a sequence of \(K\) incremental rotations onto the initial orientation \(R_{o,0}\), where each successor contact set \(c_{k+1}\) must lie in the reachable set \(\mathcal{R}(c_k, q_k)\) from the current contact set \(c_k\) and joint configuration \(q_k\), and every step must keep the estimated drop probability below a threshold \(\tau\).

$$ R_{o,T} = R(\Delta \theta_K)\cdots R(\Delta \theta_2)R(\Delta \theta_1)R_{o,0},\qquad c_{k+1} \in \mathcal{R}(c_k, q_k)\qquad \text{while}\ \Pr(\text{drop}) < \tau $$

Worked Example

# Choose the next in-hand move from orientation progress and slip risk.
moves = [
    {"name": "roll", "progress_deg": 12, "slip_risk": 0.22},
    {"name": "pivot", "progress_deg": 8, "slip_risk": 0.10},
    {"name": "finger_gait", "progress_deg": 15, "slip_risk": 0.45},
]

ranked = []
for m in moves:
    score = round(m["progress_deg"] - 20 * m["slip_risk"], 2)
    ranked.append((m["name"], score))

ranked.sort(key=lambda row: row[1], reverse=True)
print(ranked)

Expected output: The expected ranking prefers rolling because it makes strong orientation progress with moderate slip risk. In a real controller, ties would be broken using reachability of the next contact set.

Practical Recipe

1. Represent object pose in the hand frame and update it after every contact transition.
2. Log contact set, orientation residual, and slip estimate together.
3. Use short local transitions with frequent verification instead of long open-loop finger motions.
4. Reserve explicit regrasp states when the current contact family cannot reach the target orientation.
5. Evaluate on held-out object shapes, not only on one friendly benchmark object.

This section highlights the conceptual jump from grasping to dexterity. The object is no longer only constrained. It becomes a controlled body moving through a sequence of intermediate contact states.

Drawing a contact graph over reorientation states makes this concrete: dexterity is partly about planning over graph connectivity, not only about local control precision.

If orientation progress stalls, distinguish between bad state estimation, risky local transition choice, and unreachable next contact set. Each cause implies a different repair. Once these failure causes are isolated, the natural next step is to accelerate policy learning with human demonstrations, which is the subject of dexterous RL with demonstrations.