Macios Ci PostmortemSAFE

.NET for iOS, Mac Catalyst, macOS, and tvOS provide open-source bindings of the Apple SDKs for use with .NET managed languages such as C#

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: macios-ci-postmortem
description: Post-mortem analysis of CI failures across recent PRs in dotnet/macios. Identifies flaky tests, infrastructure issues, and shared regressions by analyzing builds from the last week. Files or updates GitHub issues for failures unrelated to any specific PR. Use when asked to "find flaky tests", "CI post-mortem", "what's been failing in CI", or "file issues for flaky failures".
---

# macios CI Post-Mortem

Analyze CI failures across recent PRs to identify flaky tests, infrastructure issues, and shared regressions that are not caused by any specific PR. File or update GitHub issues for these.

## References

Read these as needed during investigation:

- `references/azure-devops-cli.md` — az CLI commands, artifact naming conventions, and JSON parsing caveats.

## Overview

This skill operates in four phases:

1. **Discovery** — collect all recent PR-validation builds from AzDO
2. **Extraction** — for failed builds, extract normalized failure records
3. **Classification** — categorize failures as flaky, infrastructure, shared regression, or PR-specific
4. **Issue Actions** — propose GitHub issues, get user confirmation, then file/update

## Phase 1: Discovery — Collect Recent Builds

**Start from builds, not PRs.** This is faster, gives access to commit SHAs for rerun detection, and captures builds for PRs that may already be closed.

### Step 1.1: List recent PR-validation builds

Use the `az` CLI to get builds from the last 7 days. The macios CI runs on `devdiv.visualstudio.com/DevDiv`.

```bash
# Get the date 7 days ago in ISO format
SINCE=$(python3 -c "from datetime import datetime, timedelta; print((datetime.utcnow() - timedelta(days=7)).strftime('%Y-%m-%dT%H:%M:%SZ'))")

# List recent builds for the PR pipeline
az pipelines build list \
  --org https://devdiv.visualstudio.com \
  --project DevDiv \
  --reason pullRequest \
  --result failed \
  --top 200 \
  -o json > /tmp/postmortem_builds.json
```

Also fetch partially succeeded builds (these contain test failures):

```bash
az pipelines build list \
  --org https://devdiv.visualstudio.com \
  --project DevDiv \
  --reason pullRequest \
  --result partiallySucceeded \
  --top 200 \
  -o json > /tmp/postmortem_builds_partial.json
```

### Step 1.2: Parse and filter builds

```python
import json
from datetime import datetime, timedelta, timezone

since = datetime.now(timezone.utc) - timedelta(days=7)

def load_builds(path):
    with open(path) as f:
        content = f.read()
    return json.JSONDecoder().raw_decode(content)[0]

builds = load_builds('/tmp/postmortem_builds.json') + load_builds('/tmp/postmortem_builds_partial.json')

# Filter to last 7 days and macios pipelines
recent = []
for b in builds:
    finish = b.get('finishTime', '')
    if not finish:
        continue
    ft = datetime.fromisoformat(finish.replace('Z', '+00:00'))
    if ft < since:
        continue
    # Only include macios pipelines
    defn = b.get('definition', {}).get('name', '')
    if 'macios' not in defn.lower() and 'xamarin-macios' not in defn.lower():
        continue
    recent.append({
        'id': b['id'],
        'result': b['result'],
        'pr': b.get('triggerInfo', {}).get('pr.number', ''),
        'sourceBranch': b.get('sourceBranch', ''),
        'sourceVersion': b.get('sourceVersion', ''),  # commit SHA — critical for rerun detection
        'pipeline': defn,
        'finishTime': finish,
    })

print(f"Found {len(recent)} builds from {len(set(b['pr'] for b in recent if b['pr']))} PRs")
```

### Step 1.3: Group builds for rerun detection

Group by `(pr, pipeline, sourceVersion)`. Multiple builds with the same commit SHA for the same PR/pipeline are reruns.

```python
from collections import defaultdict

# Group: (pr, pipeline, commitSHA) -> [builds]
groups = defaultdict(list)
for b in recent:
    key = (b['pr'], b['pipeline'], b['sourceVersion'])
    groups[key].append(b)

# Also group by just (pr, pipeline) to see if new commits fixed things
pr_pipeline = defaultdict(list)
for b in recent:
    key = (b['pr'], b['pipeline'])
    pr_pipeline[key].append(b)
```

## Phase 2: Extraction — Get Failure Details

For each failed/partiallySucceeded build, extract failure information. Use a SQL database to track failures across builds.

### Step 2.1: Set up failure tracking

```sql
CREATE TABLE IF NOT EXISTS ci_failures (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    build_id INTEGER,
    pr TEXT,
    pipeline TEXT,
    commit_sha TEXT,
    finish_time TEXT,
    job_name TEXT,
    failure_type TEXT,     -- 'TestFailure', 'BuildFailure', 'TimedOut', 'Crashed', 'Infrastructure'
    test_fullname TEXT,    -- e.g. 'MonoTouchFixtures.SomeTest.TestMethod'
    platform TEXT,         -- e.g. 'ios', 'tvos', 'macos', 'maccatalyst'
    config TEXT,           -- e.g. 'Debug (ARM64)', 'Release (x64)'
    error_signature TEXT,  -- normalized error message / top stack frame
    raw_message TEXT
);
```

### Step 2.2: For each build, get the timeline and TestSummary artifacts

Only process builds with failures. For efficiency, first check the timeline for failed jobs, then only download artifacts for those jobs.

```bash
# Get timeline
az devops invoke --area build --resource timeline \
  --route-parameters project=DevDiv buildId=<buildId> \
  --org https://devdiv.visualstudio.com -o json > /tmp/timeline_<buildId>.json
```

Parse the timeline to find failed jobs:

```python
import json

with open(f'/tmp/timeline_{build_id}.json') as f:
    data = json.JSONDecoder().raw_decode(f.read())[0]

failed_jobs = []
for r in data.get('records', []):
    if r.get('type') == 'Job' and r.get('result') == 'failed':
        failed_jobs.append({
            'name': r['name'],
            'id': r['id'],
            'logId': r.get('log', {}).get('id'),
        })
```

### Step 2.3: Download TestSummary artifacts (fast triage)

TestSummary artifacts are small and quick to download. Use them first to identify which jobs failed:

```bash
arti

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (1)

SECTION II: XAMARIN STUDIO ENTERPRISE AND XAMARIN STUDIO PROFESSIONAL: When you acquire an active and valid subscription for either Microsoft Visual Studio Enterprise or Microsoft Visual Studio Profes

Gates applied: no_behavioural_pass.