Skillshare Cli E2e TestSAFE

📚 Sync skills, agents, MCP, plugins to all AI CLI tools with one command and simplify team sharing.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: skillshare-cli-e2e-test
description: >-
  Run isolated E2E tests in devcontainer from ai_docs/tests runbooks. Use this
  skill whenever the user asks to: run an E2E test, execute a test runbook,
  validate a feature end-to-end, create a new runbook, or test CLI behavior in
  isolation. If you need to run a multi-step CLI validation sequence (init →
  install → sync → verify), this is the skill — it handles ssenv isolation,
  flag verification, and structured reporting. Prefer this over ad-hoc docker
  exec sequences for any test that follows a runbook or needs reproducible
  isolation.
argument-hint: "[runbook-name | new]"
metadata: 
  targets: [claude, universal]
---

Run isolated E2E tests in devcontainer. $ARGUMENTS specifies runbook name or "new".

Before acting, run `python3 scripts/ai-context.py testing`. The topic is the source of truth for isolation, runbook quality and reporting rules; this skill retains the execution flow and mdproof recipes.

## Flow

### Phase 0: Environment Check

1. Confirm devcontainer is running and get container ID:
   ```bash
   CONTAINER=$(docker compose -f .devcontainer/docker-compose.yml ps -q skillshare-devcontainer)
   ```
   - If empty → prompt user: `docker compose -f .devcontainer/docker-compose.yml up -d`
   - Ensure `CONTAINER` is set for all subsequent `docker exec` calls.

2. Confirm Linux binary is available:
   ```bash
   docker exec $CONTAINER bash -c \
     '/workspace/.devcontainer/ensure-skillshare-linux-binary.sh && ss version'
   ```

3. Confirm mdproof is installed:
   ```bash
   docker exec $CONTAINER /workspace/.devcontainer/ensure-mdproof.sh
   ```
   This auto-installs from GitHub release, or falls back to `/workspace/bin/mdproof` (local dev binary).

4. Check for lessons learned from previous runs:
   ```bash
   test -f /workspace/.mdproof/lessons-learned.md && cat /workspace/.mdproof/lessons-learned.md
   ```
   If the file exists, read it before writing or debugging runbooks — it contains known gotchas and assertion patterns.

### Phase 1: Detect Scope

1. Preview all available runbooks via the container:
   ```bash
   docker exec $CONTAINER mdproof --dry-run --report json /workspace/ai_docs/tests/
   ```
   This returns JSON with every runbook's steps, commands, and expected assertions — no manual markdown parsing needed. Use this to understand what each runbook covers.

2. Identify recent changes (unstaged + recent commits):
   ```bash
   git diff --name-only HEAD~3
   ```
3. Match changes to relevant runbooks (compare changed file paths against step commands in the JSON output).

### Phase 2: Select Tests

Prompt user (via AskUserQuestion):

- **Option A**: Run existing runbook (list all available + mark those related to recent changes)
- **Option B**: Auto-generate new test script based on recent changes
- **Option C**: If $ARGUMENTS specifies a runbook, skip to Phase 3

### Phase 3: Prepare & Execute

#### Running existing runbook:

1. Create isolated environment with **auto-initialization**:
   ```bash
   ENV_NAME="e2e-$(date +%Y%m%d-%H%M%S)"

   # Use --init to automatically run 'ss init -g' with all targets
   docker exec $CONTAINER ssenv create "$ENV_NAME" --init
   ```

2. Execute the entire runbook via mdproof inside the container:
   ```bash
   docker exec $CONTAINER env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
     ssenv enter "$ENV_NAME" -- \
     mdproof --report json \
     /workspace/ai_docs/tests/<runbook_file>.md
   ```
   mdproof executes each step (`bash -c <command>`) in the ssenv-isolated HOME, then returns structured JSON:
   ```json
   {
     "version": "1",
     "runbook": "<runbook_file>.md",
     "duration_ms": 12345,
     "summary": { "total": 7, "passed": 5, "failed": 1, "skipped": 1 },
     "steps": [
       {
         "step": { "number": 1, "title": "...", "command": "...", "expected": ["..."] },
         "status": "passed",    // "passed" | "failed" | "skipped"
         "exit_code": 0,
         "stdout": "...",
         "stderr": "..."
       }
     ]
   }
   ```

3. Analyze the JSON output:
   - **All passed** → proceed to Phase 4
   - **Any failed** → filter for failures only (full JSON can be too large for terminal output):
     ```bash
     mdproof --report json runbook.md 2>&1 | jq '{
       summary: .summary,
       failed: [.steps[] | select(.status == "failed") | {
         step: .step.number, title: .step.title,
         exit_code: .exit_code,
         failed_assertions: [.assertions[]? | select(.matched == false) | .pattern],
         stderr: (.stderr // "" | .[0:200])
       }]
     }'
     ```
   - **Skipped steps** (executor=`manual`) → these need manual verification, run them individually:
     ```bash
     docker exec $CONTAINER env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
       ssenv enter "$ENV_NAME" -- <command from step.command>
     ```

4. For failed steps, debug individually using manual docker exec (same as before):
   ```bash
   docker exec $CONTAINER env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
     ssenv enter "$ENV_NAME" -- bash -c '<failed step command>'
   ```
   - **Prefer `--json` + `jq` for assertions** — see the JSON Reference below

#### Generating new runbook:

1. Read `git diff HEAD~3` to find changed files in `cmd/skillshare/` or `internal/`
2. Read changed files to understand new/modified functionality
3. **Validate all CLI flags before writing** — for every `ss <command> <flag>` in the runbook:
   - Grep `cmd/skillshare/<command>.go` for the exact flag string (e.g. `"--force"`)
   - Run `ss <command> --help` inside container if needed
   - Common mistakes to avoid:
     - `uninstall --yes` → **wrong**, use `--force` / `-f`
     - `init --target <name>` → **wrong**, `init` has no `--target` flag
     - `init -p` has a **completely separate flag set** from global `init` — only supports `--targets`, `--discover`, `--select`, `--mode`, `--dry-run`. Global-only flags like `--no-copy`, `--no-skill`, `--no-git`, `--all-targets`, `--force` do NOT

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (1)

.devcontainer/bin/dev-servers

Gates applied: no_behavioural_pass.