Triage Ci FlakeBLOCK

Payload is the open-source, fullstack Next.js framework, giving you instant backend superpowers. Get a full TypeScript backend and admin panel instantly. Use Payload as a headless CMS or for building powerful applications.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: triage-ci-flake
description: Use when CI tests fail on main branch after PR merge, when investigating flaky test failures, or when user provides a PR URL/number to aggregate all failing tests
allowed-tools: Write, Bash(date:*), Bash(mkdir -p *)
---

# Triage CI Failure

## Overview

Systematic workflow designed specifically for triaging and fixing end-to-end (e2e) test failures in CI, especially flaky Playwright tests that pass locally but fail in CI. E2E tests that made it to `main` are usually flaky due to timing, bundling, or environment differences.

**CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.**

## When to Use

- CI test fails on `main` branch after PR was merged
- Test passes locally but fails in CI
- Test failure labeled as "flaky" or intermittent
- E2E test timing out in CI only
- User provides a PR URL/number with failing checks

## PR-Based Workflow (When PR URL/Number Provided)

When the user provides a PR URL (e.g., `https://github.com/payloadcms/payload/pull/16701`) or PR number:

### Step 1: Fetch PR Status Checks

First, use `tool_search` with query "github pull request status checks" to load the GitHub tools.

Then use the `github-pull-request_pullRequestStatusChecks` tool to get all failing checks:

```
Tool: github-pull-request_pullRequestStatusChecks
Parameters:
  pullRequestNumber: <extracted from URL or provided>
  repo: { owner: "payloadcms", name: "payload" }
```

### Step 2: Aggregate Failing Tests

Parse the status checks response and create a summary table:

| Suite                 | Check Status | Target URL |
| --------------------- | ------------ | ---------- |
| admin**e2e**list-view | failed       | [link]     |
| plugin-import-export  | failed       | [link]     |

For each failing check, the `context` field contains the test suite name (e.g., `admin__e2e__list-view`).

### Step 3: Extract Failure Details from Each Check

For each failing check:

1. Visit the `targetUrl` to get detailed failure logs
2. Extract the specific test name and error message
3. Add to the aggregated failure list

Present a consolidated summary:

```markdown
## Failing Tests Summary

### 1. admin**e2e**list-view (3/4)

- **Test**: "should use custom pagination limit"
- **Error**: Locator `.per-page__button` not found
- **File**: test/admin/e2e/list-view/e2e.spec.ts:1495

### 2. plugin-import-export

- **Test**: "should inherit limit from list view URL"
- **Error**: Locator `.per-page__button` not found
- **File**: test/plugin-import-export/e2e.spec.ts:150
```

### Step 4: Identify Common Patterns

Look for patterns across failures:

- Same selector errors → likely a component change needing test updates
- Same test file → localized issue
- Different errors in same suite → may be test pollution

### Step 5: Proceed with Triage

For each unique failure, follow the standard reproduction workflow below.

## MANDATORY First Steps

**YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.**

1. **Extract** suite name, test name, and error from CI logs
2. **EXECUTE**: Kill port 3000 to avoid conflicts
3. **EXECUTE**: `pnpm dev $SUITE_NAME` (use run_in_background=true)
4. **EXECUTE**: Wait for server to be ready (check with curl or sleep)
5. **EXECUTE**: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE")
6. **If test passes**, **EXECUTE**: `pnpm prepare-run-test-against-prod`
7. **EXECUTE**: `pnpm dev:prod $SUITE_NAME` and run test again

**Only after EXECUTING these commands and seeing their output** can you proceed to analysis and fixes.

**"Analysis from logs" is NOT reproduction. You must RUN the commands.**

## Core Workflow

```dot
digraph triage_ci {
    "CI failure reported" [shape=box];
    "Extract details from CI logs" [shape=box];
    "Identify suite and test name" [shape=box];
    "Run dev server: pnpm dev $SUITE" [shape=box];
    "Run specific test by name" [shape=box];
    "Did test fail?" [shape=diamond];
    "Debug with dev code" [shape=box];
    "Run prepare-run-test-against-prod" [shape=box];
    "Run: pnpm dev:prod $SUITE" [shape=box];
    "Run specific test again" [shape=box];
    "Did test fail now?" [shape=diamond];
    "Debug bundling issue" [shape=box];
    "Unable to reproduce - check logs" [shape=box];
    "Fix and verify" [shape=box];

    "CI failure reported" -> "Extract details from CI logs";
    "Extract details from CI logs" -> "Identify suite and test name";
    "Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
    "Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
    "Run specific test by name" -> "Did test fail?";
    "Did test fail?" -> "Debug with dev code" [label="yes"];
    "Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
    "Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
    "Run: pnpm dev:prod $SUITE" -> "Run specific test again";
    "Run specific test again" -> "Did test fail now?";
    "Did test fail now?" -> "Debug bundling issue" [label="yes"];
    "Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
    "Debug with dev code" -> "Fix and verify";
    "Debug bundling issue" -> "Fix and verify";
}
```

## Step-by-Step Process

### 1. Extract CI Details

From CI logs or GitHub Actions URL, identify:

- **Suite name**: Directory name (e.g., `i18n`, `fields`, `lexical`)
- **Test file**: Full path (e.g., `test/i18n/e2e.spec.ts`)
- **Test name**: Exact test description
- **Error message**: Full stack trace
- **Test type**: E2E (Playwright)

### 2. Reproduce with Dev Code

**CRITICAL: Always run the specific test by name, not the full suite.**

**SERVER MANAGEMENT RULES:**

1. **ALWAYS kill all servers before starting a new one**
2. **NEVER assume ports are free**
3. **ALWAYS wait for server ready confirmation before running tests**

```bash
#

BLOCKgrade D · trust 69/100 Do not install this without reading the findings. The audit found something that could harm you or your machine.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

declared (1 observation(s))

Dependencies

pinned

Secrets in source

none-found

Findings (2)

const result = await exec(

.claude/skills

Gates applied: no_behavioural_pass.

[![ModelsAtlas audit: Triage Ci Flake](https://modelsatlas.com/skills/payloadcms/triage-ci-flake/badge.svg)](https://modelsatlas.com/skills/payloadcms/triage-ci-flake/)

Not your listing, or something out of date? Claim or update your listing.