Validate metric meaning

Data Semantics & Metric Validation

Use this skill when defining, querying, transforming, or aggregating data for charts, KPIs, and dashboards.

Core Metric Integrity Principles

1. Clarify the Question, Numerator, Denominator, and Grain First:
   • Before choosing a chart, define the exact metric formula, unit of measure, currency/scale, entity grain (per-user, per-session, per-order, per-event), and population inclusion rules.
   • Never modify an approved business metric definition autonomously to make a chart look smoother.
2. Never Average Averages or Unweighted Percentages:
   • Computing the mean of daily conversion rates or regional margins without weighting by the underlying denominator ($\sum \text{numerator}_i / \sum \text{denominator}_i$) produces mathematically false totals whenever bucket sizes differ.
   • Always carry both numerator and denominator through rollups and compute ratios at the final display grain.
3. Guard Against Simpson's Paradox & Hidden Mix Shifts:
   • Aggregate trends can reverse or mask opposing subgroup movements when cohort composition shifts. Provide cohort/category breakdowns or drill-downs alongside top-level KPIs when mix effects can mislead decisions.

Missing Data vs. Zero vs. Not Applicable

• Never Coerce null / Missing Measurements to 0:
  • A sensor outage, uncollected day, or future period is null (missing), whereas $0 revenue or 0 errors is an observed measurement.
  • Render missing intervals as explicit gaps, dashed segments, or "no data" markers—never plot them as drops to zero or silently interpolate without marking the estimate.
• Separate Measured Facts from Estimates & Forecasts:
  • Visually and textually distinguish historical actuals, incomplete current-period data ("Month-to-date, partial"), and projections/confidence intervals.

Temporal & Data Quality Hygiene

• Distinguish Event Time (occurred_at) from Ingestion Time (processed_at):
  • Know which timestamp governs the metric and display data freshness ("Last updated at…") and late-arriving data windows.
• Inspect Anomalies Before Plotting:
  • Check for duplicates, negative adjustments/refunds, extreme outliers, and currency/unit mismatches so raw ingestion artifacts are not shown as genuine spikes.

Checklist

• Are the metric formula, unit, grain, and denominator explicitly defined and traced to approved rules?
• Are rates, percentages, and averages aggregated using weighted numerators and denominators ($\sum N_i / \sum D_i$)?
• Are missing values (null), observed zeros (0), and partial current periods distinguished visually and mathematically?
• Are event timestamps, timezones, and data freshness timestamps explicitly handled?