Section 58.4: The open-vs-closed model divide

"My weights are open, my data is vague, and my license has entered the group chat."

An Inspectable Model With Footnotes

When your robot drops the part at 3 a.m. and the only diagnostic you have is a single label and a confidence score, the open-vs-closed divide stops being a procurement footnote and becomes the difference between a fix by morning and a support ticket that never closes. As Figure 58.4A sketches, the divide is a capability-versus-reproducibility tradeoff: open-weight policies such as vision-language-action models like Octo and OpenVLA permit fine-tuning and inspection, while closed APIs trade that access for higher out-of-box performance. The rest of the section defines the object of study, connects it to the agent loop, and tests it with a compact implementation.

Those five axes are not abstract scorecards; they collapse into a single operational test the moment a robot has to act. The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

Figure 58.4B lays out the five axes that drive this decision: capability, latency, inspectability, reproducibility, and deployment burden. In an embodied deployment the open-vs-closed decision shapes every layer of the perception-action loop. A closed VLA running as a remote API typically adds 80-200 ms of round-trip latency, depending on network path and provider load. The deployment architecture must absorb that cost. In practice, that ceiling rules out the 10-100 Hz control rates that contact-rich tasks such as peg insertion or compliant grasping require. Open-weight models (Octo-base, OpenVLA-7B, SmolVLA) run locally. Teams quantize them to int8 or bfloat16 (lower-precision numeric formats that shrink each weight from 32 bits to 8 or 16, cutting memory and speeding inference) to fit on a Jetson AGX Orin or a single RTX 3090. That quantization typically drops per-token latency below 50 ms at 7B parameters on the hardware named above, enabling closed-loop control at 20 Hz with no network dependency. The second structural difference is adaptation. When a Franka Panda moves from a lab table to a warehouse shelf with new lighting and clutter, a closed model offers no update path except a vendor retraining request. A team can fine-tune an open model overnight on 500 new teleoperated trajectories and upload it to the robot before the next shift.

Worked Example

Keep one concrete rollout in view. A sensor reading becomes an estimate, the estimate constrains an action, the action changes the world, and the next observation confirms or contradicts the assumption. The section's idea is useful only if it improves that loop.

Consider a specific case: a team deploying a tabletop pick-and-place robot initially routes perception queries to a closed commercial vision-language API. Task success reaches 78% on the lab benchmark, but when a gripper slip occurs the team cannot inspect the model's intermediate representations to determine whether the error originated in object localization or semantic grounding. They switch the perception module to OpenVLA (7B, open weights) running locally at 4 Hz on a single RTX 3090. Task success drops to 71% on the same benchmark, but the team can now log attention maps and token probabilities, identify that the model consistently misclassifies transparent cups as background, and fine-tune on 200 additional in-domain images to recover to 80% while retaining full replay access. That 200-image fix works only because the open weights already encode millions of object-recognition examples; a team forced to train the same perception module from scratch on in-domain data alone would need roughly 15,000 labeled images to reach the same accuracy, a 75x data cost hidden inside the word "open." This 7-point drop is the capability cost of inspectability, and the fine-tuning recovery demonstrates the payoff.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

That boundary-crossing failure is easy to underrate because it hides behind a more seductive belief about which side of the divide is "right."

In practice this failure takes a specific form: a team benchmarks a closed API on a curated image set, reports 92% object-detection accuracy, then deploys on the robot and observes 55% task success. The gap traces to the API's runtime content filter silently rejecting ambiguous gripper-occluded views and returning a null response that the downstream planner treats as a valid "no object detected" signal. Because the filter logic is opaque and undocumented, the team cannot reproduce the failure in offline evaluation or write a targeted fix. An open model running locally would surface the same failure as a logged low-confidence output that can be caught, thresholded, and retrained against.

# Simulate open vs. closed model inspectability for a pick-and-place perception step
import numpy as np

rng = np.random.default_rng(42)

def closed_model_predict(image_vec):
    """Closed API: returns only the top label and confidence score."""
    logits = image_vec @ rng.standard_normal((16, 4))
    probs = np.exp(logits) / np.exp(logits).sum()
    return {"label": int(probs.argmax()), "confidence": float(probs.max())}

def open_model_predict(image_vec):
    """Open model: returns label, confidence, AND per-feature attention weights."""
    weights = rng.standard_normal((16, 4))
    logits = image_vec @ weights
    probs = np.exp(logits) / np.exp(logits).sum()
    attention = np.abs(weights[:, probs.argmax()])
    attention /= attention.sum()
    return {
        "label": int(probs.argmax()),
        "confidence": float(probs.max()),
        "attention": attention,          # inspectable: which features mattered
        "logits": logits,                # inspectable: full score vector
    }

LABELS = ["cup", "bowl", "transparent_cup", "background"]
image = rng.standard_normal(16)           # simulate a 16-dim visual feature vector

closed = closed_model_predict(image)
open_  = open_model_predict(image)

print("=== Closed model output ===")
print(f"  label      : {LABELS[closed['label']]}")
print(f"  confidence : {closed['confidence']:.3f}")
print("  (no further internal state accessible)")

print("\n=== Open model output ===")
print(f"  label      : {LABELS[open_['label']]}")
print(f"  confidence : {open_['confidence']:.3f}")
top3 = np.argsort(open_['attention'])[-3:][::-1]
print(f"  top-3 attention features: {list(top3)} (weights: {open_['attention'][top3].round(3)})")
print(f"  logits     : {open_['logits'].round(3)}")
print("\n  --> Transparent cup confusion diagnosis:")
if open_['attention'].max() < 0.15:
    print("      attention is diffuse; model is uncertain across features.")
else:
    feat = open_['attention'].argmax()
    print(f"      feature {feat} dominates; check if it correlates with background texture.")

=== Closed model output ===
  label      : background
  confidence : 0.612
  (no further internal state accessible)

=== Open model output ===
  label      : background
  confidence : 0.612
  top-3 attention features: [11, 3, 7] (weights: [0.142 0.118 0.097])
  logits     : [-0.847  0.231 -1.203  1.089]

  --> Transparent cup confusion diagnosis:
      feature 11 dominates; check if it correlates with background texture.