Skip to main content

Remote data loader

Foxglove can load data from any queryable source using a remote data loader deployment.

To get started, build a backend which queries your data and produces MCAP on-demand. Then, deploy the remote data loader to cache and process it for visualization. Finally, connect Foxglove to the remote data loader for visualization.

Data backend​

A data backend is an HTTP service that you build that produces MCAP data on-demand. The Foxglove SDK includes utilities for building services written in Rust, Python, and C++. The data backend HTTP API consists of a manifest endpoint and one or more data endpoints.

Manifest endpoint​

The manifest endpoint provides a list of sources to load data from.

Requests include a set of user-defined query arguments, which describe the data to be visualized. Users include these query arguments in their Foxglove URL.

For example, a simple manifest endpoint might accept a single request parameter:

GET /manifest?recording=first-recording

and respond with:

{
  "name": "first-recording",
  "sources": [
    {
      "topics": [
        {
          "name": "/cos",
          "messageEncoding": "protobuf",
          "schemaId": 1
        }
      ],
      "schemas": [
        {
          "id": 1,
          "name": "point.Point",
          "encoding": "protobuf",
          "data": "CjQKC1BvaW50LnByb3RvEgVwb2ludCIWCgVQb2ludBINCgV2YWx1ZRgBIAIoAWIGcHJvdG8z"
        }
      ],
      "url": "/data?recording-id=first-recording",
      "id": "first-recording",
      "startTime": "1970-01-01T00:00:00Z",
      "endTime": "1970-01-01T00:00:10Z"
    }
  ]
}

A full explanation of manifest properties can be found in the development guide.

This endpoint also enforces your data access policy. All backend requests include a bearer token in the Authorization header. If the manifest endpoint returns 401 Unauthorized, the user is redirected to an authentication flow.

note

The manifest endpoint is the source of truth for auth. The manifest endpoint must not return a data endpoint URL if the requestor is not authorized for that source. Even if the data endpoint would deny access to that user, the remote data loader can still serve that data out of cache.

Data endpoints​

Your data endpoint(s) provide data for visualization. This endpoint serves MCAP data to the remote data loader. For details on how to construct that response, check the development guide.

Remote data loader deployment​

The remote data loader loads, caches, and merges MCAP data from your backend. You deploy it to a Kubernetes cluster using Helm. It stores cached data in a storage bucket to enable fast seeking and streaming.

Check the deployment guide for instructions to set it up.

Usage​

Foxglove users connect to the remote data loader by opening a shareable link.

The manifest query parameters are prefixed with ds. in the Foxglove URL, but this prefix is removed before forwarding.

For example, a request to:

https://app.foxglove.dev/~/view?ds=remote-data-loader&ds.dataLoaderUrl=<your remote data loader ingress>&ds.recording=first-recording

results in this manifest request:

<your manifest URL>?recording=first-recording

Build your data backend​

To get started building with remote data loaders, check out the development guide.