Ab Test PlannerCAUTION

1255 professional Agent Skills for Claude, ChatGPT, Gemini, Cursor & Codex — PRDs, postmortems, leases, medical bills, layoffs, go-bags, new countries. Plain markdown, MIT, in Anthropic's plugin directory. Plus a typed decision layer: which skill, is it safe, ship or slip. Free in-browser or npx pm-

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: ab-test-planner
description: "Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment, design an A/B test, calculate sample size, or interpret test results. Produces a complete test plan with hypothesis, variant definitions, sample size, duration estimate, guardrail metrics, and a results interpretation guide."
---

# A/B Test Planner Skill

Design experiments that produce trustworthy results — not just directional signals. Every test output includes hypothesis, success metrics, sample size, duration, and a results interpretation guide.

## Required Inputs

Work from what is given. If any of these is missing, still produce the complete output: fill each gap with a reasonable assumption marked *(assumed, confirm)* and list those assumptions at the end. Ask a question first only when the core subject itself is missing (there is nothing to write about).
- **What is being tested** (feature, UI change, copy, pricing, onboarding step)
- **Hypothesis** (or ask to help formulate one)
- **Primary metric** (conversion rate, click-through, completion rate, etc.)
- **Baseline rate** and **minimum detectable effect** (MDE)
- **Daily eligible users** (to calculate duration)

## Experiment Design Checklist

Before running any test, confirm:
- [ ] Clear hypothesis with predicted direction
- [ ] Single primary metric (plus up to 2 guardrail metrics)
- [ ] Minimum detectable effect (MDE) defined
- [ ] Sample size calculated
- [ ] Test duration estimated
- [ ] Segment isolated (no overlap with other running tests)
- [ ] Rollback plan defined

## Hypothesis Template

> "We believe that [change] will cause [primary metric] to [increase/decrease] by [X%] for [user segment], because [rationale based on data or insight]."

Never run a test without a directional hypothesis. "Let's just see what happens" is not a hypothesis.

## Sample Size Calculator Logic

Use this formula (provide the output, not the formula, to the user):

- **Baseline conversion rate:** Current rate of primary metric
- **MDE:** Smallest change worth detecting (recommend 10–20% relative lift for most features)
- **Statistical power:** 80% (standard)
- **Significance level:** 95% (p < 0.05)

For common scenarios, provide pre-calculated estimates:

| Baseline Rate | MDE (Relative) | Required Sample per Variant |
|---|---|---|
| 5% | 20% | ~19,000 |
| 10% | 15% | ~14,000 |
| 20% | 10% | ~15,000 |
| 40% | 10% | ~9,500 |
| 60% | 5% | ~42,000 |

Always warn: "These are estimates. Use a tool like Evan Miller's calculator or Statsig for precision."

## Test Duration Guidance

Minimum: 2 full weeks (to capture weekly seasonality)
Maximum: 4 weeks (novelty effect distorts results beyond this)

`Duration = Required sample ÷ (Daily traffic × % exposed)`

Flag if traffic is too low to reach significance in under 8 weeks — recommend a different approach (e.g., holdout test, qualitative research).

## Output Format

### A/B Test Plan — [Test Name] — [Date]

**Hypothesis:**
> [Filled hypothesis template]

**Variants:**
- Control (A): [Current experience]
- Treatment (B): [Changed experience — be specific]

**Primary Metric:** [Metric name + how measured]
**Guardrail Metrics:** [Metrics that must not degrade]

**Target Segment:** [Who sees the test — % of traffic, user type]
**Traffic Split:** [50/50 recommended unless ramp-up needed]

**Sample Size Required:** ~[N] users per variant
**Estimated Duration:** [X] weeks (based on [Y] daily eligible users)
**Significance Threshold:** 95% confidence, 80% power

**Exclusions:** [Any user segments to exclude and why]

**Rollback Trigger:** If [guardrail metric] degrades by [X%], stop the test immediately.

**Results Interpretation Guide:**
- ✅ Ship if: Treatment shows [X%]+ lift on primary metric at 95% confidence AND guardrail metrics are stable
- 🔄 Iterate if: Direction is positive but not significant — consider extending or redesigning
- ❌ Reject if: No lift or negative direction at significance
- ⚠️ Inconclusive: Do not ship. Do not call it a win.

---

## Guidelines

- Always recommend against peeking at results before the test reaches planned sample size — explain p-hacking risk
- If user wants to test multiple variants, explain the multiple comparisons problem and recommend a Bonferroni correction or a Bayesian approach
- If traffic is very low (<1,000 users/day), recommend qualitative alternatives: moderated testing, 5-second tests, or user interviews
- Never approve a test with no guardrail metrics — always protect revenue, retention, or core engagement

## Anti-Patterns

- [ ] Do not run a test without a directional hypothesis — "let's see what happens" produces uninterpretable results
- [ ] Do not declare a winner before reaching the pre-planned sample size — peeking at results inflates false positive rates
- [ ] Do not test multiple independent changes in a single variant — you won't know which change caused the result
- [ ] Do not use engagement metrics (clicks, time-on-page) as the primary metric when the goal is revenue or retention — proxy metrics mislead
- [ ] Do not ignore guardrail metrics — a conversion lift that causes a support ticket spike is not a win

## Deeper Materials

This skill ships with support files — use them when they are available:

- **`references/test-validity-traps.md`** — The Validity Traps That Quietly Invalidate A/B Tests. Apply it while producing the output; it carries the calibration and judgment calls the method summary above compresses.
- **`templates/test-plan.md`** — a fill-in version of the deliverable with the quality gates inline. Offer it when the user wants to work the document themselves rather than have it generated.

## Scoring Rubric (0–40)

Score any output of this skill before handing it over; 32+ is ship-quality.

| Dimension | 0 | 5 | 10 |
|---|---|---|---|
| Statistical rigour | No sample size, or a number with no stated baseline/MDE behind it | S

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (5)

templates/pm-discovery-agent/skills/assumption-mapper/SKILL.md

templates/pm-discovery-agent/skills/discovery-interview-guide/SKILL.md

templates/pm-discovery-agent/skills/job-story-mapper/SKILL.md

templates/pm-discovery-agent/skills/user-interview-synthesis/SKILL.md

templates/pm-launch-agent/skills/content-calendar/SKILL.md

Gates applied: no_behavioural_pass.