Skip to main content

Remote data loader

Foxglove can load data from any queryable source using a remote data loader deployment.

To get started, build a backend which queries your data and produces MCAP on-demand. Then, deploy the remote data loader to cache and process it for visualization. Finally, connect Foxglove to the remote data loader for visualization.

Data backend​

A data backend is an HTTP service that you build that produces MCAP data on-demand. The Foxglove SDK includes utilities for building services written in Rust, Python, and C++. The data backend HTTP API consists of a manifest endpoint and one or more data endpoints.

Manifest endpoint​

The manifest endpoint provides a list of sources to load data from.

Requests include a set of user-defined query arguments, which describe the data to be visualized. Users include these query arguments in their Foxglove URL.

For example, a simple manifest endpoint might accept a single request parameter:

GET /manifest?recording=first-recording

and respond with:

{
  "name": "first-recording",
  "sources": [
    {
      "topics": [
        {
          "name": "/cos",
          "messageEncoding": "protobuf",
          "schemaId": 1
        }
      ],
      "schemas": [
        {
          "id": 1,
          "name": "point.Point",
          "encoding": "protobuf",
          "data": "CjQKC1BvaW50LnByb3RvEgVwb2ludCIWCgVQb2ludBINCgV2YWx1ZRgBIAIoAWIGcHJvdG8z"
        }
      ],
      "url": "/data?recording-id=first-recording",
      "id": "first-recording",
      "startTime": "1970-01-01T00:00:00Z",
      "endTime": "1970-01-01T00:00:10Z"
    }
  ]
}

A full explanation of manifest properties can be found in the development guide.

This endpoint also enforces your data access policy. All backend requests include a bearer token in the Authorization header. If the manifest endpoint returns 401 Unauthorized, the user is redirected to an authentication flow.

note

The manifest endpoint is the source of truth for authorization. The manifest endpoint must not return a data endpoint URL if the requestor is not authorized for that source. Even if the data endpoint would deny access to that user, the remote data loader can still serve that data out of cache.

If manifest response caching is enabled, cached manifest responses also cache authorization decisions until the configured time-to-live expires.

Data endpoints​

Your data endpoint(s) provide data for visualization. This endpoint serves MCAP data to the remote data loader. For details on how to construct that response, check the development guide.

Remote data loader deployment​

The remote data loader loads, caches, and merges MCAP data from your backend. You deploy it to a Kubernetes cluster using Helm. It stores cached data in a storage bucket to enable fast seeking and streaming.

Check the deployment guide for instructions to set it up.

Usage​

Foxglove users connect to the remote data loader by opening a shareable link.

Put manifest parameters in ds.manifestParams as an encoded query string. Foxglove forwards those parameters to your manifest endpoint.

To open multiple sources, repeat ds.manifestParams. They merge onto one timeline by default; set ds.compare=true to compare them. Provide a single ds.dataLoaderUrl; it applies to every manifest.

For example, a request to:

https://app.foxglove.dev/~/view?ds=remote-data-loader&ds.dataLoaderUrl=<your remote data loader ingress>&ds.manifestParams=recording%3Dfirst-recording

results in this manifest request:

<your manifest URL>?recording=first-recording

To compare two recordings from the same loader:

https://app.foxglove.dev/~/view?ds=remote-data-loader&ds.dataLoaderUrl=<your remote data loader ingress>&ds.compare=true&ds.manifestParams=recording%3Dfirst-recording&ds.manifestParams=recording%3Dsecond-recording

Metadata sent to Foxglove​

The Foxglove app makes one request to Foxglove's control plane when you open a remote data loader: a profile lookup to confirm your organization's plan supports the feature. This request doesn't include the data loader URL, manifest parameters, authentication tokens, or any data from your backend.

Your data backend, manifest responses, MCAP content, and authentication all stay between your browser, your remote data loader deployment, and your data backend. None of it reaches Foxglove's control plane.

note

The layout you use to visualize this data is a separate concern. If you save that layout, its configuration, including topic and field names, is stored in Foxglove's control plane.

Build your data backend​

To get started building with remote data loaders, check out the development guide.