Skip to main content

Build Golden Signals

This playbook describes how to build golden‑signal dashboards and alerts for a service using Unified Monitoring.

Prerequisites​

  • The service is represented in CMDB with clear application, service, environment, and owner attributes.
  • Metrics from cloud providers and/or Breeze/Zabbix are available for the service.

Steps​

  1. Identify critical user journeys
    List the main ways users interact with the service (for example, API calls, web transactions, batch jobs).

  2. Define metrics for each golden signal

    • Latency: response time for key endpoints or operations.
    • Traffic: request rate, throughput, or job count.
    • Errors: error rate or failure count.
    • Saturation: CPU, memory, queue depth, or connection utilization.

  3. Build a service dashboard

    • Create a dashboard filtered to the service using CMDB labels.
    • Add widgets that show each golden signal with recent history and current value.
    • Include dependency health (databases, queues, external APIs).

  4. Create alert policies

    • Define thresholds or SLO‑based alerts for each golden signal.
    • Use different severities for early warnings vs. critical impact.
    • Route alerts to the owning team’s on‑call rotation.

  5. Review and iterate

    • After incidents, review which metrics were most useful.
    • Tune thresholds and dashboards based on real‑world behavior.
    • Retire metrics and widgets that are no longer needed.