Product analytics and experiments

Product Analytics and Experiments

Measure what informs a decision, and say plainly what is assumed.

Metrics

• Start from the decision and the questions; every metric and event must change what someone does.
• One primary metric for the goal, a few input metrics the team can move, and guardrails that must not get worse (errors, support tickets, refunds, cancellations, performance, unsubscribes). A win that breaks a guardrail is not a win.
• A north star reflects value customers get, not revenue alone, and predicts retention.
• Define each metric exactly: event, population, time window, how it is computed. "Active" needs a definition.
• Targets need a baseline. When there is none, write "assumed target" and how it will be checked; never present a guessed number as data.
• Read distributions, segments and cohorts, not only averages; compare against the previous period and the target.

Tracking plan

• Per event: name (object_action, lowercase, e.g. signup_completed), trigger, properties, and the question it answers.
• Context goes into properties, not into new event names. No personal data in properties; respect consent.
• Numerators and denominators use the same closed time window; recent cohorts still inside their window are pending, not failed.
• Compute totals and rates from row-level data and cross-check them a second way.

Experiment design

• Hypothesis: "Because [evidence], we believe [change] will cause [outcome] for [audience]; we will know when [metric] moves by [amount]."
• One meaningful change per variant. Decide in advance: primary metric, guardrails, minimum effect worth detecting, significance level (usually 0.05) and power (usually 80%).
• Compute the sample per variant with a calculator from the baseline rate and the minimum effect; small effects on low baselines need very large samples.
• Duration = needed sample ÷ eligible daily traffic per variant. Run at least one full week (two business cycles in B2B), at most about 4–8 weeks. If it is infeasible, raise the minimum effect, test fewer variants, test further upstream, or decide without a test.
• Randomize per user and keep each user in one variant. Start small (90/10) when the change is risky.

Running and reading

• No peeking: do not stop for a win before the planned sample; stop early only when a guardrail is clearly harmed. Never extend a finished test to chase significance. Repeated looks need a sequential method chosen up front.
• Check the split matches the plan (a mismatch means broken randomization) and watch for novelty effects.
• Report effect size with its confidence interval, the p-value, guardrails and planned segments. A p-value is not the probability that the variant is better; significance alone does not justify shipping.
• Decide: clear win with healthy guardrails, ship; win with a guardrail concern, investigate; no difference, report the interval and stop; loss, keep the control and learn why. Unplanned segment findings are exploratory.

Before you finish

• Each metric is defined with its window and population; guardrails are named.
• Assumed targets and assumptions are marked as such.
• Sample size, duration and stopping rule were set before starting.
• Results report effect, uncertainty and guardrails, not only significance.