Create a data loader with Rust

Enterprise

Data loaders are an experimental Extension API that allow you to support more file formats in Foxglove. They are built as a Foxglove Extension using WASM.

Using Rust or C++, write a small WASM binary that runs in your browser alongside the Foxglove app. When a file using your extension is opened, your WASM binary will be used to translate your file format into something that Foxglove can understand.

info

Data loaders are currently in beta. Contact us at [email protected] for early access.

This guide walks through the process of creating a data loader using Rust. It frequently references the Rust Data Loader API reference docs.

Lifecycle of a data loader

The process begins when the Foxglove app loads your extension and registers the data loader.

When the user opens files in your format, Foxglove will instantiate your data loader with a list of paths. It will then call initialize. Your implementation should use the Reader interface provided to gather summary information about those files to return as an Initialization.

This summary information includes:

• The topic names and channels in your file(s), and their schema definitions
• The start and end time of your file(s)
• Other supplementary information, such as the number of messages on each channel
• Any "problems" such as errors or warnings the user should be aware of regarding the file

With the data loader initialized for the provided file(s), Foxglove will now ask your data loader to return instances of the MessageIterator to populate panels with data.

These iterators may be read to completion to populate plots or state transition panels, or kept open for the duration of playback. Foxglove will create as many message iterators as it needs to satisfy the layout.

Setting up

To get started, open up a new terminal window and cd into the directory where your source code will live.

Clone the repository and copy the rust-data-loader-template folder to scaffold out your data loader.

# Clone the extension repository
git clone https://github.com/foxglove/create-foxglove-extension.git
# Copy the template and rename with your data loader name
cp -r ./create-foxglove-extension/examples/rust-data-loader-template ./my-data-loader

Navigate to the my-data-loader directory, and inspect the contents to see the following layout:

• rust: a directory containing the Rust source code implementing the DataLoader trait.
• src/index.ts: a lightweight Foxglove extension that loads the compiled Rust WASM and registers it with the Foxglove app
• config.ts: custom configuration for the extension build, that allows the WASM to be bundled
• package.json: the manifest for the extension containing the build script, and extension name and dependencies

Implement DataLoader and MessageIterator traits

The Rust data loader template contains an implementation of the DataLoader and MessageIterator traits that return errors when called.

Each method must be updated to contain the logic for your data loader and your specific file format. For more information on what these methods need to do, view the docs for each at the foxglove_data_loader API reference. For a complete example implementation, view the ndjson-data-loader example from the create-foxglove-extension repo.

Update the extension

In order to tell Foxglove about your data loader, you need to call the registerDataLoader extension API.

This is done in the src/index.ts file. Update this file to include the correct file type for your data loader.

import { Experimental } from "@foxglove/extension";

// Import the .wasm file as a base64 data URL to be bundled with the extension
import wasmUrl from "../rust/target/wasm32-unknown-unknown/release/foxglove_data_loader.wasm";

export function activate(extensionContext: Experimental.ExtensionContext): void {
  extensionContext.registerDataLoader({
    type: "file",
    wasmUrl,
    // update this to be your file format:
    supportedFileType: ".xyz",
  });
}

By default, the extension will be called foxglove-data-loader-template. Update package.json to give your data loader a better name.

Build the extension

Once you've completed implementing your data loader, it is time to build and package the extension. In order to do this, you'll need to have a WASM target installed for Rust.

Install wasm32-unknown-unknown using rustup:

rustup add target wasm32-unknown-unknown

The Foxglove app also supports the wasm32-wasip1 target. Use this if your data loader requires a more complicated build.

With WASM support, finally install dependencies and package your extension:

npm install # install dependencies
npm run package # build the extension and package

Once this completes, you'll have a .foxe Foxglove extension ready to install in the app.

Install the extension, and start visualizing your custom file formats!

Advanced Concepts

Backfill

Backfill is a Foxglove concept that ensures panels have data when seeking.

When seeking to a point in time, Foxglove will ask your data loader to give it the latest messages at that point in time. These messages will be used to populate panels before playback begins, to make sure that the data is up-to-date.

By default, data loaders return no backfill data. To enable it implement the get_backfill method on your trait:

impl DataLoader for MyLoader {
    fn get_backfill(
        &mut self, args: BackfillArgs
    ) -> Result<Vec<Message>, Self::Error> {
        // return the latest message for each channel
    }
}

Since backfill runs when a user seeks to a new location on the playback slider, it is important it runs quickly to avoid delays from buffering. Consider limiting the amount of data searched to get the latest messages.

Multiple files

Data loaders may support loading multiple files of the same extension at once.

If your data loader has been written to support this, set supportsMultiFile to true during registration:

export function activate(extensionContext): void {
    extensionContext.registerDataLoader({
        type: "file",
        wasmUrl,
        supportedFileType: ".xyz",
        supportsMultiFile: true
    });
}