A B Test DesignSAFE

Designer Skills Collection: agentic skills, commands, and plugins for design — from research to systems, UI, interaction, and delivery.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: a-b-test-design
description: Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
---
# A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
## What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
## Test Structure
### 1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
### 2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
### 3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
### 4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
### 5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
### 6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
## Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
## When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
## Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.

[![ModelsAtlas audit: A B Test Design](https://modelsatlas.com/skills/owl-listener/a-b-test-design/badge.svg)](https://modelsatlas.com/skills/owl-listener/a-b-test-design/)

Not your listing, or something out of date? Claim or update your listing.