Skip to main content

Performance & Scale

Use this guide to learn how to optimize Analytics dataset size, streamline transformations, and schedule jobs effectively to avoid hitting CRM Analytics limits or slowing down dashboards.

Data Optimization​

Filter and shape data as early as possible:

  • Apply filters at the recipe input stage to exclude irrelevant records (for example, deleted or historical objects that are not needed for analytics).
  • Drop unused fields and normalize names in Data Governance so datasets stay lean and readable.
  • Aggregate where possible (for example, daily per‑service metrics instead of raw events) and keep detailed history only where it is truly needed.

See also: Data Preparation

Dataset Design and Granularity​

  • Use different datasets for different subject areas (adoption, reliability, compliance, cost) instead of one oversized dataset.
  • Separate raw vs analytics‑ready datasets:
    • Raw/landing datasets at high granularity.
    • Aggregated datasets that back most dashboards and KPIs.
  • Choose grains that match the questions you need to answer (for example, per‑user‑per‑day vs per‑event).

Processing Optimization​

Recipe and pipeline efficiency:

  • Schedule heavier recipes during off‑peak hours where possible.
  • Minimize the number of nodes and combine related transformations into a smaller number of Transform steps.
  • Prefer recipes over legacy dataflows, and reuse shared building‑block recipes (for example, shared compute assets) instead of duplicating logic.
  • Use incremental refresh patterns (where available) so that only new/changed data is processed.

Scheduling and Concurrency​

  • Avoid scheduling multiple heavy recipes at the exact same time, especially if they share sources or targets.
  • Align schedules with upstream data arrival (for example, BigQuery billing exports, Compliance Engine outputs) so downstream datasets are refreshed after inputs are ready.
  • Document and review the recipe schedule matrix periodically to ensure it still matches business needs.

Dashboard and Query Patterns​

  • Prefer a small number of well‑designed dashboards over many similar variants; use filters to switch between scopes instead of cloning dashboards per team or app.
  • Limit the number of heavy widgets (for example, long time‑range charts at fine granularity) on a single dashboard.
  • Use global filters and shorter default time windows; let users widen the range only when needed.
  • Keep SAQL customizations focused and avoid very complex expressions when standard queries are sufficient.

Monitoring Usage and Jobs​

  • Regularly review Usage in Data Manager to monitor:
    • Dataset row counts and growth trends.
    • External data uploads (for example, BigQuery imports) relative to limits in Limits & Quotas.
  • Use Jobs Monitor to:
    • Track failed or long‑running recipes.
    • Identify recipes that frequently rerun or reprocess large volumes of unchanged data.
  • Consider building an internal “Data Pipelines Health” dashboard that surfaces recipe/job status and dataset freshness.

Retention and Archival​

  • Define retention expectations per dataset (for example, 13 months of daily metrics vs 3 years of monthly rollups).
  • Snapshot detailed data into summarized datasets (for example, monthly snapshots) and retire raw history once it is no longer required for analytics or audit.
  • For very long‑term history or cross‑domain analysis, rely on warehouse/BI exports described under Integrations – Data Warehouse instead of keeping everything in CRM Analytics.

Security Predicates and Performance​

  • Keep row‑level security predicates as simple as possible and align them with existing CMDB ownership and access models.
  • Avoid stacking many overlapping predicates or highly complex logical expressions on a single dataset, as they can slow down queries.
  • Where appropriate, push scoping logic into recipes (for example, pre‑segmented datasets per business unit) rather than relying solely on runtime filters.