Skip to main content

Monitoring

Monitor your self-hosted Primary Site using Prometheus.

Configure Prometheus​

The Primary Site services expose various application metrics that Prometheus can scrape and ingest.

The first option is to use one of the major cloud providers' managed integrations:

Alternatively, deploy Prometheus to your cluster directly using the directions below.

Create Helm repositories and namespace​

Add Prometheus Helm repositories:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create a Kubernetes namespace for Prometheus:

kubectl create namespace prometheus

Install Prometheus​

Install Prometheus to your Kubernetes cluster:

helm install prometheus prometheus-community/prometheus -n prometheus

Configure pods for scraping​

Locate the queryService, siteController, and inboxListener sections in your values.yaml file.

For each section, add a pod annotation under deployment instructing Prometheus to scrape the pod:

siteController:
  deployment:
    podAnnotations:
      prometheus.io/scrape: true

Upgrade your deployment with the new configuration. You may need to alter this command with your own namespace, Helm release name, and file path:

helm upgrade primary-site -f values.yaml --namespace foxglove

Prometheus will now scrape the metrics endpoints of the deployed services. To confirm this is working, forward the port of the Prometheus UI to view collected metrics:

kubectl -n prometheus port-forward service/prometheus-server 9090:80

Visit http://localhost:9090 and enter the query {app="site-controller"}. Executing that should show a list of metrics scraped from the edge controller.

Install custom metrics adapter​

Finally, install the Prometheus adapter for the Kubernetes custom metrics API using Helm.

This assumes you have installed Prometheus into the "prometheus" namespace. If you are using a different namespace, replace the second component of the URL accordingly.

helm install prometheus-custom-metrics-adapter prometheus-community/prometheus-adapter -n prometheus --set prometheus.url=http://prometheus-server.prometheus.svc.cluster.local

After a couple minutes, you should see custom metrics:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

If the output contains metrics, you are ready to create an autoscaler on custom metrics.

Monitor metrics​

Once Prometheus is configured, the services expose useful application metrics for monitoring.

Inbox listener​

note

This only applies to the query-optimized storage mode.

  • import_success_count - Number of imports successfully processed (counter)
  • import_quarantine_count - Number of imports quarantined (counter)
  • import_input_size_bytes - Size of input files in bytes (histogram)
  • import_output_size_bytes - Size of output files (processed data files) in bytes (histogram)
  • import_processing_time_seconds - Processing time for imports in seconds (histogram)
  • output_file_count - Number of output files per import (histogram)
  • input_message_count - Number of messages per import (histogram)

Indexer​

note

This only applies to the index-in-place storage mode.

  • build_manifest_duration_ms - Time spent indexing recordings (histogram)
  • recording_bytes_read - Recording data read by the indexer (counter)
  • recording_bytes_total - Total size of indexed recordings(counter)
  • recording_count - Total number of indexed recordings (counter)
  • recording_size_gib - Recording sizes in GiB (histogram)
  • upload_manifest_duration_ms - Time spent uploading recording manifests (histogram)

Site controller​

  • unleased_pending_import_count - Number of backlogged pending imports for processing (gauge)
  • oldest_unprocessed_pending_import_age_secs - Age of oldest unprocessed pending import in seconds (gauge)

Query service​

The query service serves stream and search requests. The application metrics below are listed without a prefix, which is what a default install emits: queryService.deployment.metrics.namespace and queryService.deployment.metrics.subsystem are both empty in values.yaml. Setting either prepends it, giving <namespace>_<subsystem>_stream_duration_ms. Standard process metrics (process_cpu_seconds_total, process_resident_memory_bytes, and so on) are served from the same endpoint and are never prefixed.

Requests

  • in_flight_requests - HTTP requests currently being served (gauge)
  • successful_stream_requests - Stream requests served successfully (counter)
  • failed_stream_requests - Stream requests that failed with an internal error (counter)
  • successful_search_requests - Search requests served successfully (counter)
  • failed_search_requests - Search requests that failed with an internal error (counter)
  • total_bytes_emitted - Bytes written to clients across all stream requests (counter)
  • total_bytes_read_from_object_storage - Bytes read from object storage across all requests (counter)

Stream requests

  • time_to_first_message_ms - Time between reception of a request and the first message written out (histogram)
  • time_to_first_byte_ms - Time between reception of a request and the first byte written to the response (histogram)
  • stream_duration_ms - Total wall-clock duration of a stream request (histogram)
  • stream_response_bytes - Bytes written to the client per stream request (histogram)
  • stream_input_files - Number of files merged to produce the output stream (histogram)
  • files_request_latency_ms - Time to request the list of data files from the Foxglove API (histogram)

Output throughput is recorded both overall and split by how many input files the stream merged, which makes it possible to tell a slow large merge apart from a slow small one:

  • output_bytes_per_second_all - Output throughput for all streams (histogram)
  • output_bytes_per_second_le_10_input_files - Output throughput for streams with 10 or fewer inputs (histogram)
  • output_bytes_per_second_le_100_input_files - Output throughput for streams with 100 or fewer inputs (histogram)
  • output_bytes_per_second_le_1000_input_files - Output throughput for streams with 1000 or fewer inputs (histogram)
  • output_bytes_per_second_gt_1000_input_files - Output throughput for streams with more than 1000 inputs (histogram)

Search requests

  • search_duration_ms - Total wall-clock duration of a search request (histogram)
  • search_response_bytes - Bytes written to the client per search request (histogram)
  • search_rows_returned - Number of hit rows returned per search request (histogram)
  • search_bytes_read_from_object_storage - Bytes read from object storage per search request (histogram)