Skip to main content

Search configuration

Search runs on the query service in your Primary Site. Depending on the amount of data being searched, your query service might need more resources than the defaults provide.

This page covers three common adjustments:

  • Running the query service on a higher-tier instance
  • Increasing the ingress timeout so long searches don't get cut off
  • Giving the query service more memory

Run the query service on a higher-tier instance​

If you're experiencing slow queries, the query service pods may not have enough CPU or memory on the nodes they're scheduled on. Search benefits from machines with more CPU and memory than the defaults in your cluster.

Use queryService.deployment.nodeSelector in your Helm values file to target a specific node pool or compute class. The exact label depends on your cloud provider.

On GKE, use the cloud.google.com/machine-family node selector to target a higher-tier machine family.

queryService:
  deployment:
    nodeSelector:
      cloud.google.com/machine-family: c4
      # Or n4 or n2 for a more cost-effective option.

Increase the ingress timeout​

If you're experiencing timeouts for your searches, the ingress controller is likely closing the connection before the query service returns its response.

Ingress controllers typically apply a default read timeout of 30 to 60 seconds, and searches that scan a large dataset can take longer than this.

Raise the timeout on your ingress using controller-specific annotations. The examples below set a 1 hour (3600 second) timeout.

The GKE Ingress controller reads timeout settings from a BackendConfig custom resource. Reference the BackendConfig from the query service with the cloud.google.com/backend-config annotation.

queryService:
  service:
    annotations:
      cloud.google.com/backend-config: '{"default": "query-service-backend-config"}'

Create the BackendConfig in the same namespace as your Primary Site:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: query-service-backend-config
  namespace: foxglove
spec:
  timeoutSec: 3600
tip

If you use a self-hosted ingress controller instead of a cloud-native one, set the equivalent timeout for your controller. See the relevant configuration reference for NGINX, Traefik, or HAProxy.

Choose a timeout that covers your longest expected search. Setting the timeout too low will cause searches over large datasets to fail.

Increase query service memory​

If you're experiencing dropped requests during search, the query service may be running out of memory. Search holds recording metadata in memory while evaluating a query, and if the query service exceeds its memory limit, Kubernetes will terminate the pod and in-flight searches will fail.

Raise the query service's memory request and limit in your Helm values file. We recommend starting at 8Gi and increasing further if you still see pods terminated for exceeding their limit.

queryService:
  deployment:
    resources:
      requests:
        cpu: "2"
        memory: 8Gi
      limits:
        memory: 8Gi

Scale the query service horizontally​

For sites that handle many concurrent searches, combine the memory increase above with horizontal scaling so that concurrent searches are spread across multiple pods rather than competing for memory on a single one. See Autoscaling for HPA configuration that targets the query service.

Apply the changes​

After updating your values file, upgrade your release to apply the changes:

helm upgrade foxglove-primary-site foxglove/primary-site \
    --values ./values.yaml \
    --namespace foxglove