Inspect Robots

Inspect Robots is an evaluation framework for robot policies: you define a task, an embodiment, and a policy, and it runs trials and records an EvalLog. Dropbear plugs in as a policy, so a Dropbear-hosted model can be evaluated by the same harness, on the same task and embodiment, as any policy you already run. The adapter is inspect-robots-dropbear (Apache 2.0). It changes one thing in your evaluation: which policy is selected. Your registered task and embodiment stay exactly as they are. Runnable reference code for everything below lives in examples/: a complete evaluation script and a YAM embodiment skeleton, in two variants — native-30hz/ and reduced-15hz/ — both adapted from runs that completed against production. Copy from there rather than retyping this guide.

This guide covers the software path. It does not authorize an unattended physical run. For any physical robot test you own the validated limits, an attended operator gate, a working e-stop, and a rehearsed termination procedure. Serving evidence is not safety evidence.

Before you start

You need an Inspect Robots evaluation that already runs on your hardware with some other policy — a scripted or teleop baseline is ideal. Bring up the robot, the cameras, and the task first, and confirm trials complete end to end. Adding Dropbear to an evaluation that is not yet working makes the first failure much harder to attribute. You also need a Dropbear API key and an entitlement for the model you intend to call. Some models, including DreamZero-YAM, are granted per organization.

1. Install

uv add inspect-robots-dropbear

The adapter pins the exact Dropbear SDK release it was qualified against, so it installs a matching dropbear[dreamzero] for you. Confirm the policy registered without opening a cloud session:

inspect-robots list policies

The output must contain dropbear.

2. Authenticate

dropbear login

A robot controller usually has no browser, so pass the key directly there instead:

dropbear login --api-key "<your key>"

Both paths write the same credential file, and both are verified the same way:

dropbear status --model dreamzero-yam

status confirms the credential is accepted and reports whether the model has capacity, before you spend a session finding out.

3. Select the policy

Change only the policy selection in your existing evaluation command:

inspect-robots run \
  --task your-task \
  --embodiment your-yam \
  --policy dropbear -P model=dreamzero-yam \
  --log-dir logs

Substitute your own registered --task and --embodiment; --policy dropbear -P model=dreamzero-yam is the entire edit. Everything else — epochs, scorer, logging — is unchanged.

Parameters

Parameter	Default	Meaning
`model`	required	Dropbear model id, for example `dreamzero-yam`
`sampling`	`async_latest`	Action sampling mode, see below
`control_hz`	model’s native rate	Rate the action chunk is executed at, 5–30 Hz
`keep_warm_s`	`0`	Hold the session warm after close so the next run skips the cold start, 0–3600 s. Billed while held
`startup_timeout_s`	`1800`	Budget for the first connection while the model loads and warms up
`timeout_s`	`60`	Per-step action deadline

async_latest is the qualified default: inference is single-flight and latest-only, so the robot always acts on the freshest available chunk. Use async_8 only as an explicit compatibility or rollback path, and upstream_eval only for an agreed open-loop dataset evaluation — it blocks at each replan and is not intended for hardware. The first connection is slow on purpose. A cold model loads weights and warms up before it serves, which is why startup_timeout_s defaults to 30 minutes. Later trials in the same run reuse the open connection.

Skipping the cold start while you iterate

A cold start is roughly two minutes, paid once per connection — every trial in one evaluation shares it. It only hurts when you are iterating: fixing a camera mapping, re-running, fixing the next thing, and paying two minutes each time. keep_warm_s holds the session after close so the next run reclaims it instead of starting cold:

--policy dropbear -P model=dreamzero-yam -P keep_warm_s=300

Measured against production, back to back:

	cold	reclaimed
Time to first action	147 s	23 s

The reclaim is matched on your organization, user, model, region mode, and optimization settings, so a later inspect-robots run with the same policy arguments picks up the session you left warm.

A warm hold is billed at the full rate, and close() no longer stops the meter. With keep_warm_s set, closing parks the session rather than terminating it: the GPU stays reserved for you, and billing continues until the window expires or you reclaim it. In a verified run, a session that was active for well under a minute billed 141.9 s across its park.So this is a cost you are choosing, not a cache. At DreamZero-YAM’s rate a five-minute hold is worth it only if you actually re-run inside it — otherwise you have paid five minutes to save two. Leave it at 0 for unattended or production runs, and reach for it only during hands-on iteration.

Control rate, and who owns the clock

DreamZero-YAM emits 24 actions per chunk at a native 30 Hz. control_hz sets the rate you execute them at — any whole rate from 5 to 30:

--policy dropbear -P model=dreamzero-yam -P control_hz=15

The whole chunk is executed at the rate you choose, so it stretches in time rather than being resampled: 0.8 s at 30 Hz, 1.6 s at 15 Hz. Nothing is dropped or interpolated, so the trajectory the robot executes is the one the model emitted, just played slower. Nothing server-side takes a rate; this is entirely a client-side decision.

control_hz does not make anything run at that rate. Inspect’s rollout imposes no wall-clock pacing, so the real rate is however fast your embodiment’s step() returns — hardware paces itself on its actuators, and anything simulated or stubbed must pace itself deliberately. Your embodiment is the clock.What control_hz changes is what everything else believes: the action scheduler sizes its replan threshold from it (a 400 ms round trip is 12 steps of a 24-step chunk at 30 Hz, but 6 at 15 Hz), the compatibility check reconciles it against your embodiment’s declared rate, and it is stamped on every action in the EvalLog. Declare the rate you actually achieve — a wrong one degrades replanning while the run still looks healthy.A rate disagreement between policy and embodiment is only a warning, never an error, so it will not stop a run. The adapter measures the real gap between policy steps, records it as step_interval_ms in the sidecar, and warns once if the measured rate is more than 25% from the commanded one.

4. Run the evaluation

What your embodiment must provide

On every policy step, the task and embodiment must supply:

top_cam, left_cam, and right_cam uint8 images, each with a real monotonic capture time;
finite packed joint_pos state with shape (14,), ordered left arm, left gripper, right arm, right gripper; and
Inspect’s integer extra["env_step"], starting at zero and advancing once per delivered action.

The adapter declares a 14-dimensional raw absolute-joint action at the commanded rate and returns exactly one action per act() call, while Dropbear owns the action buffering behind it. See Model contracts for the full DreamZero-YAM contract, including gripper conventions. Capture times must be real. The model reasons over a short history of frames, so a fabricated or stale clock silently degrades its output rather than raising.

The whole script

The shape of a full run, with the hardware halves stubbed. The complete versions are run_eval.py and yam_embodiment.py.

import collections, json, pathlib, time

import numpy as np
from inspect_robots import Observation, eval as run_evaluation
from inspect_robots.embodiment import EmbodimentBase, EmbodimentInfo, StepResult
from inspect_robots.registry import resolve

LOG_DIR = "logs"


class YamEmbodiment(EmbodimentBase):
    """Your arm. Replace the driver calls; the rest is the contract above."""

    @property
    def info(self) -> EmbodimentInfo:
        ...  # 14-D joint_pos action, three 360x640 cameras, control_hz=30.0

    def _observation(self, instruction: str) -> Observation:
        top, left, right, times = self._robot.frames()  # your driver
        return Observation(
            images={"top_cam": top, "left_cam": left, "right_cam": right},
            # Seconds off a real capture clock, keyed per camera.
            image_times=dict(zip(("top_cam", "left_cam", "right_cam"), times)),
            state={"joint_pos": self._robot.state.astype(np.float64)},
            state_time=time.monotonic_ns() / 1e9,
            instruction=instruction,
            extra={"env_step": self._env_step},  # advances once per action
        )

    def step(self, action) -> StepResult:
        payload = getattr(action, "data", action)  # an Action, not a bare array
        self._robot.execute(np.asarray(payload).reshape(-1))  # your driver
        self._env_step += 1
        return StepResult(
            observation=self._observation(""),
            reward=0.0,
            terminated=False,
            truncated=False,
            info={},
        )


# `resolve` returns a constructed object, not a factory, and takes the same
# parameters you would pass with `-P`.
policy = resolve("policy", "dropbear", model="dreamzero-yam")
task = resolve("task", "your-task")
embodiment = resolve("embodiment", "your-yam")

try:
    logs = run_evaluation(task=task, policy=policy, embodiment=embodiment, log_dir=LOG_DIR)
finally:
    # Sessions bill while open and hold an exclusive lease, so this is not
    # optional — including when the run raised.
    policy.close()

# Then check where the actions actually came from. See the warning below.
sidecar = sorted(pathlib.Path(LOG_DIR).glob("dropbear/*/*.jsonl"))[-1]
rows = [json.loads(line) for line in sidecar.read_text().splitlines() if line.strip()]
print(collections.Counter(row["action_source"] for row in rows))

Closing

Confirm nothing is still running:

dropbear sessions list

The adapter registers a bounded process-exit fallback, but it is not a substitute for the explicit close(), and it cannot help if the machine loses power. dropbear sessions stop --all clears a zombie lease.

If you construct the policy object yourself rather than naming it in the registry, you own its lifecycle. Inspect closes what it resolved from a registry name; it does not close objects you passed in. The script above resolves the policy itself, so the explicit close() is required.

5. Diagnostics

The adapter writes one atomic sidecar per trial and records its relative path at TrialRecord.metadata["dropbear_telemetry"]:

dropbear/<run_id>/<sanitized-scene-id>-e<epoch>.jsonl

Each row carries serving identity, the source control tick, camera capture-to-execution age, timing, chunk and merge disposition, and the same Inspect environment step. Join to the EvalLog, the action JSONL, or a Rerun timeline on env_step; use join_key for Dropbear chunk-level diagnostics. Sidecars never contain action vectors, images, credentials, or endpoints, so they are safe to attach to a bug report. The two numbers worth reading first are the observation-to-action round trip and action_source, which records for every step whether the robot executed a model action or held its previous position.

A green evaluation can contain no model actions at all. Inspect reports success when trials complete without raising, and episode_length counts steps, not inference. A run where every step held position looks identical in the EvalLog to one the model drove throughout.Count action_source before trusting a result — the last few lines of the script above. Seeing only hold means the model never contributed, and no amount of green in the EvalLog says otherwise. The usual cause is a control loop that runs faster than one round trip, so no chunk can arrive before the episode ends — most often an embodiment that returns instantly instead of pacing itself at the control rate the way hardware does.

A high but non-zero hold fraction means the action buffer drained, usually because the round trip exceeded the chunk horizon. Lowering the command rate lengthens the horizon and can absorb it — but only if your embodiment actually slows to match. Changing the parameter alone just makes the scheduler wrong. Two production runs from Sydney, same code and same input, show the size of the effect:

	30 Hz	15 Hz
Chunk horizon	0.8 s	1.6 s
Holds	81 / 120 (67.5%)	17 / 600 (2.8%)
GPU inference p50	333 ms	321 ms
Observation-to-action p50	644 ms	594 ms

Latency barely moved — that is distance and GPU, and the rate touches neither. What changed is that a chunk now covers 1.6 s of wall clock against a ~600 ms round trip, so the buffer stops draining. Note the cost: the arm sees a fresh observation half as often, which matters for contact-rich or fast motions. Treat a high hold fraction as the reason to drop the rate, not as the default. Once the round trip exceeds the chunk horizon, the hold fraction does not degrade gracefully — it goes to 100%. No chunk can ever land before the buffer it was meant to refill has drained, so every step holds and the model contributes nothing. Re-running the same 30 Hz configuration from a tethered mobile connection instead of fixed broadband, with nothing else changed, the round trip rose from 644 ms to 945 ms — past the 800 ms horizon — and model-sourced actions went from 39/120 to 0/120. The run still reported success. That makes the horizon a threshold worth knowing:

rate	chunk horizon	round trip must stay under
30 Hz	0.8 s	~800 ms
15 Hz	1.6 s	~1.6 s

The margin at 30 Hz is thin if you are far from the serving region: a few hundred milliseconds of extra round trip is the difference between a working run and an empty one. Anything shared, metered, or wireless — a hotspot, a busy lab network, a VPN — can cost you that much on its own. Check action_source rather than trusting the status, and prefer 15 Hz when the margin is not comfortable.

Where the boundaries are

Inspect Robots remains canonical for the EvalLog, scores, commanded-action logs, stored frames, Rerun recordings, and trial termination state. Dropbear is responsible only for turning an observation into an action chunk. The adapter calls neither your embodiment nor your hardware directly. Compatibility with this contract means the cameras, state, action, clock, and rate line up. It does not by itself establish physics parity, task success, or physical-robot safety — those remain yours to demonstrate on your own hardware.

Getting started

Build with Python

Robot guides

Reference

Help

Before you start

1. Install

2. Authenticate

3. Select the policy

Parameters

Skipping the cold start while you iterate

Control rate, and who owns the clock

4. Run the evaluation

What your embodiment must provide

The whole script

Closing

5. Diagnostics

Where the boundaries are

​Before you start

​1. Install

​2. Authenticate

​3. Select the policy

​Parameters

​Skipping the cold start while you iterate

​Control rate, and who owns the clock

​4. Run the evaluation

​What your embodiment must provide

​The whole script

​Closing

​5. Diagnostics

​Where the boundaries are

Before you start

1. Install

2. Authenticate

3. Select the policy

Parameters

Skipping the cold start while you iterate

Control rate, and who owns the clock

4. Run the evaluation

What your embodiment must provide

The whole script

Closing

5. Diagnostics

Where the boundaries are