Build Golden Signals

This playbook describes how to build golden‑signal dashboards and alerts for a service using Unified Monitoring.

Prerequisites

• The service is represented in CMDB with clear application, service, environment, and owner attributes.
• Metrics from cloud providers and/or Breeze/Zabbix are available for the service.

Steps

1. Identify critical user journeys
   List the main ways users interact with the service (for example, API calls, web transactions, batch jobs).

2. Define metrics for each golden signal

   • Latency: response time for key endpoints or operations.
   • Traffic: request rate, throughput, or job count.
   • Errors: error rate or failure count.
   • Saturation: CPU, memory, queue depth, or connection utilization.

3. Build a service dashboard

   • Create a dashboard filtered to the service using CMDB labels.
   • Add widgets that show each golden signal with recent history and current value.
   • Include dependency health (databases, queues, external APIs).

4. Create alert policies

   • Define thresholds or SLO‑based alerts for each golden signal.
   • Use different severities for early warnings vs. critical impact.
   • Route alerts to the owning team’s on‑call rotation.

5. Review and iterate

   • After incidents, review which metrics were most useful.
   • Tune thresholds and dashboards based on real‑world behavior.
   • Retire metrics and widgets that are no longer needed.