Label episodes

Store labels as episode properties. Set them in Foxglove or write results back from a labeling tool. Properties can hold labels directly or link to annotation files.

Define episode properties

Before anyone can set a label, an organization admin defines the property it belongs to. In Settings → Episodes, click Add property and give the property a label, a key, and a type.

Choose Single-line text, Multi-line text, Boolean, Number, Single-select, or Multi-select. For example, use Single-select with success and failure for an outcome. The key identifies the property in filters, searches, and code. You can't change the key or type after saving.

To link an episode to an annotation file, define Single-line text properties such as annotations_uri and annotations_format. A Single-line text value can be up to 1,024 characters long, and a Multi-line text value up to 2,048. Some characters, such as emoji, count as two.

To define properties from a script, call the Python client's create_custom_property method with the episode resource type, or use the Custom Properties API.

Label episodes in Foxglove

Any member of your organization can set property values on the latest version of an episode. There are three places to do it:

• On an episode's page, open the Properties tab and edit the values there.
• In Edit episode, open the Properties tab. This is convenient when you are already trimming the episode.
• To label many episodes at once, select them on the Episodes page or in a dataset's draft and click Edit properties in the toolbar above the list.

In the Edit properties dialog, choose a Property, an Action such as Set value or Remove property, and a value. Click Add change to change another property in the same step, then click Apply.

Each change saves a new version of the episode. When you label from a dataset's draft, the draft uses the new versions right away. Labels you set anywhere else reach a dataset when you update its draft to the latest episode versions.

Label episodes with Automations

Automations can give each episode a first pass of labels, sometimes called silver standard labels: quick, rule-based descriptions from the rules you write in the automation's prompt. An AI agent applies them to each episode and saves what it finds as properties, so you can filter and search on them like any label you set by hand. A person can review and correct them, or refine them later into gold standard labels through human review or a labeling tool.

Give the agent somewhere to write by defining the properties first. A Multi-line text property such as summary holds a short description. A Single-select property such as activity, with options like pick, regrasp, and idle, holds the category your rules choose.

The agent looks at the same data you would. It lists the project's episodes along with each one's time range and properties. For each episode, it can open the recordings and read topic messages between the episode's start and end. It can also list the events in that window. Your rules name the topics and event types to check and say which property values to set.

Create the automation in the episodes' project with an On a schedule trigger. With a prompt like this one, each run describes only the episodes that don't have a summary yet:

List the episodes in this project that have no summary value. For each one, open its recordings and read /gripper/state and /arm/joint_states between the episode's start and end times, and list the grasp_failure events in that window. Set activity to regrasp if the gripper closes more than once or a grasp_failure event falls in the window, to pick if it closes once, and to idle if it never closes. Set summary to two sentences about what the arm did, with timestamps. If the data doesn't fit a rule, leave activity unset and explain why in summary.

Each change saves a new version of the episode, as an edit in the app does. The run can also bring those versions into a dataset's draft. Review and correct the properties there like any other label before you commit the draft yourself. An automation that creates episodes can set the same properties when it creates them.

Label episodes with an external tool

Use an external tool for bounding boxes, segmentation masks, or keypoints. Keep its annotation files in your storage and save their URIs as episode properties.

The workflow is: send episodes to the tool, label them, write the results back as properties, and commit a new dataset version.

Send episodes to a labeling tool

For a single batch, download the committed version you want to label with the Foxglove CLI:

foxglove datasets download dset_123 --version 2 --output ./to-label --include-properties

The export contains one MCAP file per episode and a manifest with episode IDs, pinned versions, and properties. Keep each episode's ID and version with its labels so you can write results back to the right episode.

This function lists a committed version's episodes, skips those with annotations, and downloads the camera topic from each remaining episode as MCAP. It uploads the file with the episode ID and version.

The ID and version also form an idempotency key. If your tool honors it, retries won't create duplicate labeling tasks. Replace the example URL and header with your tool's API.

import os

import requests
from foxglove.client import Client

client = Client(token=os.environ["FOXGLOVE_API_TOKEN"])


def send_for_labeling(dataset_id: str, version_number: int):
    members = client.get_dataset_version_episodes(
        dataset_id=dataset_id, version_number=version_number, include_properties=True
    )
    for member in members.auto_paging_iter():
        episode = member["episode"]
        if "annotations_uri" in episode.get("properties", {}):
            continue
        data = client.download_data(
            episode_id=episode["id"],
            episode_version=episode["version"],
            topics=["/camera/front"],
        )
        task_key = f"{episode['id']}:{episode['version']}"
        response = requests.post(
            "https://labeling.example.com/tasks",
            headers={"Idempotency-Key": task_key},
            files={"file": (f"{episode['id']}.mcap", data)},
            data={"episode_id": episode["id"], "episode_version": episode["version"]},
            timeout=60,
        )
        response.raise_for_status()

Send episodes when a dataset version is committed

Subscribe to the dataset.version.committed webhook to label each committed version. Its payload includes the dataset ID and version number.

Delivery is at least once. Timeouts and non-2xx responses are retried up to 5 times, with increasing delays starting at about a second. Deduplicate notifications using the payload's webhookId and webhookEventId: record the pair after a successful run and skip pairs already recorded.

Your endpoint must respond within 5 seconds, and a labeling run takes longer than that. Check the request's signature first: the fg-webhook-signature header is a hex SHA-256 HMAC of the raw request body, keyed with your webhook's token. Then put the payload on a job queue and return a 2xx status. Call handle_webhook from the worker that reads the queue.

handled_events: set[tuple[str, str]] = set()


def handle_webhook(payload: dict):
    event = (payload["webhookId"], payload["webhookEventId"])
    if payload["type"] != "dataset.version.committed" or event in handled_events:
        return
    dataset = payload["dataset"]
    send_for_labeling(dataset["id"], dataset["version"])
    handled_events.add(event)

In a service, store handled event pairs in a database so they survive restarts. Failed runs leave the pair unrecorded and can be retried. Each episode version keeps the same upload idempotency key across commits and retries.

To send every new episode as soon as it exists, subscribe to episode.created and read the episode's ID and version from its payload.

Write the labels back

Write the results back as properties. This function takes one result per episode and saves its outcome, annotation URI, and format. A call can update up to 2,000 episodes.

def write_labels(results: list[dict]):
    return client.update_episodes(
        episodes=[
            {
                "id": result["episode_id"],
                "applies_to_version": result["episode_version"],
                "properties": {
                    "outcome": result["outcome"],
                    "annotations_uri": result["annotations_uri"],
                    "annotations_format": result["annotations_format"],
                },
            }
            for result in results
        ]
    )


updated = write_labels(
    [
        {
            "episode_id": "ep_123",
            "episode_version": 3,
            "outcome": "success",
            "annotations_uri": "s3://my-labels/towel-folding/ep_123/v3/coco.json",
            "annotations_format": "coco",
        }
    ]
)

Pass the version you sent to the tool as applies_to_version. If someone has edited an episode since you exported it, the call fails and changes nothing, so stale labels never overwrite newer work. Properties you leave out keep their values.

Update the dataset and commit

Labels create new episode versions. Update the draft to use them, then commit it.

client.update_dataset_episodes(
    dataset_id="dset_123",
    update=[
        {"id": episode["id"], "version": episode["version"]} for episode in updated
    ],
)
client.commit_dataset(dataset_id="dset_123", description="Outcome labels, batch 12")

You can do the same step in the app by selecting the episodes in the dataset's draft and clicking Update to latest version. Then commit the draft.

Use labels in training

Read episode properties alongside messages to use labels as training targets or skip rejected episodes. For annotation files, read the URI from the properties and load the file with your storage client.