ab-test-analysis
Analyzes A/B test results: statistical significance, sample size checks, confidence intervals and a ship, extend or stop recommendation.
- Kind
- Skill
- Works with
- Claude Code, Cursor
- Source
- PM Skills (Pawel Huryn) @ 2a12295a7717
- Path
- pm-data-analytics/skills/ab-test-analysis/SKILL.md
- Jobs
- Testing
- Topics
- Data analysis & BI
Credit
- Made by
- phuryn
- Original
- github.com/phuryn/pm-skills/blob/2a12295a7717569baa7be181b4cf…
- License
- MIT
Listed by Skill Manifest; not made by us.
InstallInstall it from a plugin marketplace.
claude plugin install pm-data-analytics@pm-skills
Install factsScanned 2 Nov 2026
- Adds
- .claude/skills/ab-test-analysis/ (the skill folder)
- Runs scripts
- None found
- Uses the network
- None found
- Needs secrets
- None found
- Source revision
- 2a12295a7717
- Licence
- MIT
- Last activity
- 29 Oct 2026
- Review
- Reviewed
Found by a static scan of the stored text; nothing is run. “None found” is not a guarantee, so read what you install before you run it.
- Only the main file is read; other files in the folder are not scanned
## A/B Test Analysis
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
### Context
You are analyzing A/B test results for **$ARGUMENTS**.
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
### Instructions
1. **Understand the experiment**:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
2. **Validate the test setup**:
- **Sample size**: Is the sample large enough for the expected effect size?
- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
- Flag if the test is underpowered (<80% power)
- **Duration**: Did the test run for at least 1-2 full business cycles?
- **Randomization**: Any evidence of sample ratio mismatch (SRM)?
- **Novelty/primacy effects**: Was there enough time to wash out initial behavior changes?
3. **Calculate statistical significance**:
- **Conversion rate** for control and variant
…