Skip to main content

Query service configuration

Configure the query service​

The query service handles the querying playback of recordings uploaded to your Primary Site.

You can configure compute and memory, and scale horizontally (see Autoscaling). In most cases, the default configuration does not need to be changed.

Buffered Object Limit​

During a request, the query service merges multiple cloud storage objects into a streaming HTTP response, buffering objects that will be required soon so data is available as quickly as possible. The BUFFERED_OBJECT_LIMIT environment variable caps how many objects are buffered concurrently.

This value defaults to 1024. If you're experiencing slow downloads when merging many recordings, you may benefit from lowering this limit.

Configure the new limit in your helm values file:

queryService:
  deployment:
    env:
      - name: BUFFERED_OBJECT_LIMIT
        # Defaults to 1024
        value: "512"

Topic search performance​

Topic and message-path searches read and merge a topic's files in parallel, so search speed scales with the CPU available to the query service. If searches are slow, raising the CPU requests and limits is the most effective change you can make. The search partition count defaults to the CPU limit, so adding cores automatically increases parallelism.

By default the query service requests 1 CPU and can burst to a 4 CPU limit when the node has spare capacity, but only the request is guaranteed. For consistently fast searches, raise both values:

queryService:
  deployment:
    resources:
      requests:
        cpu: "16"
      limits:
        cpu: "16"

The CPU-derived defaults work well in most cases. For finer control, use the overrides below.

Override: partition count​

Searches split a topic's files across partitions that are read from object storage and merged in parallel. Each partition can keep a CPU core busy, so the partition count should match the number of available cores.

The partition count is controlled by queryService.deployment.topicExec.partitionCount. When left empty (the default), it follows the query service's CPU limit, rounded up to whole cores. Set it explicitly to override:

queryService:
  deployment:
    topicExec:
      # Defaults to the CPU limit rounded up to whole cores
      partitionCount: "4"
Override: per-partition read concurrency​

Each partition bounds how many objects it reads from storage concurrently, controlled by queryService.deployment.topicExec.bufferedObjectLimit (default 8).

Raising it can speed up searches at the cost of more storage connections, memory, and open files per partition. This is separate from the Buffered Object Limit above, which applies to streaming playback rather than topic search.

queryService:
  deployment:
    topicExec:
      # Max concurrent object reads buffered per partition (default 8)
      bufferedObjectLimit: 16

Open file limits​

Reading many files in parallel raises the number of files the query service has open at once. Some platforms (notably AWS Fargate) ship a low open-file (nofile) soft limit that can be exhausted under load, causing Too many open files (EMFILE) errors. To prevent this, the chart raises the soft limit to 65535 on every Primary Site pod that supports it (query service, indexer, inbox listener, site controller, and garbage collector), clamped to the container's hard limit. This default should be more than enough.

If you need to constrain it, set globals.maxOpenFiles to a lower value, or to an empty string to leave the platform default untouched:

globals:
  # Open-file (RLIMIT_NOFILE) soft limit on supported pods,
  # clamped to the container hard limit. Defaults to 65535.
  maxOpenFiles: 4096

Plan Logging​

The query service merges input files according to an internal plan data structure, which is constructed at the start of every request.

If you experience degraded performance, you can set ENABLE_PLAN_LOGGING to true in your deployment environment, which includes plan information in request logs. These logs can be shared with Foxglove to troubleshoot performance issues.

Turn on plan logging by updating your helm values file:

queryService:
  deployment:
    env:
      - name: ENABLE_PLAN_LOGGING
        value: "true"