Section 60.4: Research-seminar track

"We discussed the paper for ninety minutes and left with one experiment worth running."

A Research Seminar With Standards

A seminar can spend ninety minutes admiring a robot folding laundry on video and walk out knowing nothing it can defend, or it can walk out with one experiment that would prove the demo real or expose it as a single unreplicated lab kitchen; the difference is whether anyone in the room asked which artifact backs the number. This section develops that habit into a usable mental model: first we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation. Figure 60.4A sketches the weekly rhythm that carries this habit: one paper read deeply, one presented critique, and one open question that seeds the next selection.

As a track, this format runs one session per week for a full semester (typically 10-13 weeks): each week pairs one primary paper with one artifact audit and closes with one forward question, so that by the end of term the seminar has produced a semester-long ledger of teach-now, replicate-now, and watch-only verdicts rather than a stack of disconnected summaries.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

Keep one concrete rollout in view. A sensor reading becomes an estimate, the estimate constrains an action, the action changes the world, and the next observation confirms or contradicts the assumption. The section's idea is useful only if it improves that loop.

Consider a specific case: a seminar week centered on the RT-2 paper (Brohan et al., 2023). The paper reports a 62% success rate on novel-concept tasks versus 32% for the non-language-conditioned baseline. The student presenting the mechanism explains that the vision-language-action model maps RGB frames and a text command to 8-DoF (Degrees of Freedom) robot actions at 3 Hz through a co-fine-tuned PaLM-E backbone (PaLM-E is a large vision-language model; "co-fine-tuned" means it is trained jointly on web-scale image-text data and robot action data rather than on robot data alone, so it keeps its general visual reasoning while learning to output actions). The student auditing the evidence then identifies a specific gap. All 23 evaluation tasks ran on a single Everyday Robot platform in one lab kitchen, so the cross-embodiment and cross-environment generalization claim rests on zero independent replications. The session closes with one tractable forward question: what is the smallest scene-perturbation (lighting, object position, background clutter) that collapses the success rate below the baseline? That question becomes the seed for week 8's paper selection.

Before reading the recipe below, pause and predict: if you ran the RT-2 evaluation on the same robot but shifted ambient lighting by 200 lux, would success drop by 5 percentage points, 20, or more than 40? Your answer determines which step in the recipe below matters most to you.

Practical Recipe

1. Write the sensor modalities, action space, and success metric before choosing a policy architecture. For a manipulation paper: name the RGB or depth source (e.g., wrist-mounted RealSense D405 at 30 Hz), the action representation (joint angles vs. end-effector delta, 6-DoF or 7-DoF), and the metric (pick success at 15 cm placement tolerance, averaged across 50 trials).
2. Build a scripted or teleoperated baseline first. On a Franka Panda or a MuJoCo simulation of it, a hand-coded pick-and-place script exposes the exact failure modes (grasp slip at <0.3 N grip force, Inverse Kinematics (IK) singularity near joint limits) that the learned policy must improve on.
3. Add the pretrained policy (e.g., an OpenVLA or Octo checkpoint loaded via from_pretrained) only after the scripted baseline behavior and its failure modes are fully documented, so that any regression introduced by the policy is immediately attributable.
4. Record failures as structured cases tied to the agent loop: exteroceptive error (depth dropout, occlusion), proprioceptive error (encoder slip, forward-kinematics (FK, the map from joint angles to end-effector pose) mismatch), planning error (out-of-distribution language command), actuation error (torque saturation, velocity-limit clamp), or evaluation error (marker mis-placed, human reset inconsistency).
5. Run at least one sim-to-real perturbation before trusting a result: change the object texture or mass in simulation, or shift ambient illumination by 200 lux on hardware, and report whether the success rate stays within 5 percentage points of the nominal condition.

def validate_item(payload: dict[str, object]) -> dict[str, object]:
    assert payload, "payload must not be empty"
    return payload

# Seminar ledger item.
item = {
    "paper": "foundation policy for mobile manipulation",
    "verdict": "replicate-now",
    "missing_evidence": "independent evaluation on shifted embodiments",
    "student_owner": "week_7_pair",
}
print(validate_item(item))

{'paper': 'foundation policy for mobile manipulation', 'verdict': 'replicate-now', 'missing_evidence': 'independent evaluation on shifted embodiments', 'student_owner': 'week_7_pair'}