Access and export datasets

The Foxglove SDK has readers for PyTorch and Ray Data, so to train with either one, use Train models. For labeling, evaluation, analysis, or training with a tool or package that doesn't have a Foxglove integration, use this page to read the same dataset versions in code or export them as MCAP files.

Read a dataset version in code

A dataset version pins one version of each of its episodes, so code that reads it gets the same data every time, even after someone edits an episode.

Read from Python

From Python, use the Foxglove Python client, which is documented with the Foxglove SDK. Install the foxglove-client package, version 0.24 or later. With it you can list the episodes in any dataset version along with their properties, and stream each episode's messages in log-time order, already decoded. The same client can also download a dataset version as files, or create and commit datasets from a script.

import os

from foxglove.client import Client

client = Client(token=os.environ["FOXGLOVE_API_TOKEN"])

members = client.get_dataset_version_episodes(
    dataset_id="dset_123", version_number=3, include_properties=True
)
for member in members.auto_paging_iter():
    episode = member["episode"]
    print(episode["id"], episode["version"], episode.get("properties"))

    for schema, channel, message, decoded in client.iter_messages(
        episode_id=episode["id"],
        episode_version=episode["version"],
        topics=["/joint_states"],
    ):
        print(channel.topic, message.log_time, decoded)

Each entry names an episode and the episode version that the dataset version pins. Pass that episode version when you stream messages, so you read exactly what the dataset version recorded.

JSON messages need no extra packages, and Protobuf, ROS 1, and ROS 2 messages need mcap-protobuf-support, mcap-ros1-support, or mcap-ros2-support.

To read part of a version, pass properties_query when you list its episodes. It takes the same terms as the Properties filter in the app.

Read from any other language

From any other language, use the Foxglove API, the HTTP API that the Python client is built on. It gives you the same access to datasets: list the episodes in a dataset version along with the episode version each one pins, then request each episode's data at that episode version, so you read exactly what the dataset version recorded. The response links to an MCAP file, which you can read with any MCAP library.

In shell scripts, the Foxglove CLI lists a dataset version's episodes and downloads dataset versions as files.

Export a dataset version

Export a dataset version when you want its data as files, for example to give a committed dataset version to a tool that doesn't have a Foxglove integration, or to work with it offline. You get one MCAP file for each episode, holding the episode's time range with the data from all of its recordings, and a manifest.json that records which dataset version and episode versions the files came from. If an episode can't be downloaded, the manifest lists it with status "failed" and a reason.

towel-folding-v3/
├── manifest.json
├── episode_0000_ep_123.mcap
├── episode_0001_ep_124.mcap
└── episode_0002_ep_125.mcap

The CLI and the Python client write these files into a directory, and the browser download puts them in one zip archive. Read the manifest describes each field.

Download in the browser

If your organization allows downloads from the app, click Use for training on a dataset's page and open the Browser tab. It downloads the version as a single zip archive, with the MCAP files and the manifest in one folder. Keep the browser tab open until the download finishes.

Download with the CLI

The Foxglove CLI downloads a version into a directory. Dataset downloads need version 2.2.0 of the CLI or later.

foxglove datasets download dset_123 --version 3 --output ./towel-folding-v3

Add --topics with a comma-separated list to download only those topics, and --include-properties to write each episode's properties into the manifest. On a dataset's page, the CLI tab of the Use for training dialog builds this command for the version and topics you select.

If a download stops partway, run the same command again. The CLI keeps the files that finished and downloads the rest.

Download with Python

The Python client writes the same files into a new or empty directory:

client.download_dataset(
    dataset_id="dset_123",
    version_number=3,
    output_directory="towel-folding-v3",
    include_properties=True,
)

The CLI, the Python client, and the Browser tab can also download the draft, by its version number. Its episodes can change until it is committed, so download a committed version when you need files that stay the same. With the CLI, download a changed draft into a new --output directory, because files for episodes that left the draft stay in the earlier one.

Read the manifest

The manifest is a JSON object with these fields:

Field	Contents
formatVersion	The version of the manifest format, which is 1
generatedAt	When the manifest was written, as an RFC 3339 UTC timestamp
dataset	The dataset's id, name, and projectId
version	The versionNumber and the time it was committed, as committedAt. A download of the draft has uncommitted set to true in its place.
selection	The episodeCount of the version, and the topics you selected, if any
episodes	One entry for each episode in the version

Each entry in episodes has these fields:

Field	Contents
index	The episode's position in the export, which is also the number in its file name
id	The episode's ID
version	The episode version that the dataset version pins
startTime, endTime	The episode's time range as RFC 3339 UTC strings, copied from the API with up to nanosecond precision.
status	downloaded or failed
file, byteSize	For a downloaded episode, the path of its MCAP file and the file's size in bytes. The CLI also lists them on a failed episode when it keeps a file from an earlier run.
reason	For a failed episode, why the download failed
properties	The properties of the pinned episode version, with --include-properties (CLI) or include_properties=True (Python). The browser download always includes them.

The file path differs between tools. The CLI and the browser download put the name of the output directory before the file name, as in towel-folding-v3/episode_0000_ep_123.mcap. The Python client writes the file name alone, which is relative to manifest.json.

When you run the CLI with --include-attachments=false, it records includeAttachments as false in selection.

Keep the manifest with the files, because it records which dataset version and episode versions they came from.

Export a single episode

Where downloads from the app are allowed, an episode's page has a Download button. Its dialog lets you narrow the time range and choose topics, and saves one MCAP or ROS 1 bag file.

To export one episode with the CLI, use the export command with the episode's ID:

foxglove export --episode-id ep_123 --output-file episode.mcap