Manage data

Start processing data recordings with your Foxglove Agent.

Adding recording files

Move your completed recordings into the storage directory. The Agent will watch for recordings in this directory.

The Agent watches the storage directory recursively — recordings in nested subdirectories are detected automatically, so you can organize recordings into directories (for example, one directory per mission). Each recording is identified by its path relative to the storage directory, so subdirectory names are preserved in the recording name in Foxglove.

By default, the Agent registers for native filesystem notifications when files under the storage directory are created, deleted, or modified. Since these notifications are best-effort, the Agent also periodically scans the directory to compensate for missed notifications.

tip

In environments where the storage directory has very frequent filesystem activity, the Agent may exhibit high CPU utilization and/or log throttled warnings about dropped filesystem notifications. In such environments, it may be preferable to disable filesystem notifications by setting WATCH_USE_NOTIFY=false, and rely exclusively on periodic scanning. The scan interval can be configured with WATCH_RECONCILIATION_INTERVAL, which is set to 30s by default.

When the Agent detects a new file under the storage directory, it will attempt to parse it as a recording and summarize its contents. This work is wasted for files that are not recordings, or for files that are incomplete recordings. If the storage directory contains such files, you should configure recording filters so that the Agent will ignore them. For example, you may write recordings with a special suffix like .active to indicate they are incomplete, and then rename them (using mv rather than cp) to .bag or .mcap once they are complete.

note

The Agent will ignore MCAP files that do not contain trailing magic bytes, under the assumption that the file is incomplete. It is still preferable to write incomplete files with a suffix that will be ignored, and rename when the file is complete.

note

The ROS 1 bag writer uses an .active suffix on incomplete files by default. The Agent ignores files with this suffix by default.

Import to cloud

Recordings for your device will initially appear in the Recordings page. In order to visualize and download this data, you can select these recordings and click "Import". Once the import status is displayed as "complete", the recording is available. You can also do this with the Recordings API.

Automatically import recording files

The Agent can be configured to automatically upload recording files that match a configured glob pattern. Set the WATCH_AUTO_IMPORT_PATTERN environment variable to a valid glob pattern to enable this functionality. The pattern is matched against each recording's path relative to STORAGE_ROOT, using globset syntax. Note that * matches within a single path segment, while ** matches recursively across subdirectories.

To automatically import every MCAP file the Agent detects — whether it sits directly in the storage directory or in a nested subdirectory — use a recursive ** pattern:

WATCH_AUTO_IMPORT_PATTERN="**/*.mcap"

To restrict automatic import to part of the directory tree, anchor the pattern to a subdirectory. For example, the following will cause MCAP files anywhere under the auto_import/ subdirectory of STORAGE_ROOT to be imported as soon as they are created:

WATCH_AUTO_IMPORT_PATTERN="auto_import/**/*.mcap"

Recordings that don't match the pattern are still detected and listed on the Recordings page, where you can import them on demand.

Delete recording files

You can delete recording files as needed to manage disk space on your device. Files that were imported will remain available in Foxglove after the original is deleted. Files that were not imported before deletion will no longer appear in the Recordings page.

You can also configure a retention period to automatically remove old files. The retainRecordingsSeconds property specifies the maximum age of recordings stored on the device, based on the creation time of the recording. (If set to zero, recordings are retained indefinitely.)