Gamma Incident RunbookSAFE

Model-agnostic agent-skills platform with a harness-free canonical layer, verified adapters, and the ccpi package manager. Explore at tonsofskills.com.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: gamma-incident-runbook
description: 'Manage incident response for Gamma integration issues.

  Use when experiencing production incidents, outages,

  or need systematic troubleshooting procedures.

  Trigger with phrases like "gamma incident", "gamma outage",

  "gamma down", "gamma emergency", "gamma runbook".

  '
allowed-tools: Read, Write, Edit, Bash(curl:*), Grep
version: 1.13.0
license: MIT
author: Jeremy Longshore <[email protected]>
tags:
- saas
- gamma
- incident-response
compatibility: Designed for Claude Code
---
# Gamma Incident Runbook

## Instructions

Assign severity and an opaque incident ID, pause unsafe publishing/sharing, classify the issue, apply the smallest safe mitigation, verify recovery with synthetic content, and record the owner’s decision.

## Output

Produce a redacted incident receipt with severity, impact, mitigation, recovery verification, rollback decision, owner, and follow-ups.

## Error Handling

Revoke scoped credentials on suspected exposure, do not replay content actions until access/destination checks pass, and keep private content out of incident artifacts.

## Examples

During a synthetic sharing outage, pause the staging integration, confirm no duplicate publish occurs, restore one canary, and resume only after the incident owner records recovery evidence.

## Overview

Systematic procedures for responding to and resolving Gamma integration incidents.

## Prerequisites

- Access to monitoring dashboards
- Access to application logs
- On-call responsibilities defined
- Communication channels established

## Incident Severity Levels

| Level | Description | Response Time | Escalation |
|-------|-------------|---------------|------------|
| P1 | Complete outage, no presentations | < 15 min | Immediate |
| P2 | Degraded, slow or partial failures | < 30 min | 1 hour |
| P3 | Minor issues, workaround available | < 2 hours | 4 hours |
| P4 | Cosmetic or non-urgent | < 24 hours | None |

## Quick Diagnostics

### Step 1: Check Gamma Status

```bash
set -euo pipefail
# Check Gamma status page
curl -s https://status.gamma.app/api/v2/status.json | jq '.status'

# Check our integration health
curl -s https://your-app.com/health/gamma | jq '.'

# Quick connectivity test
curl -w "\nTime: %{time_total}s\n" \
  -H "Authorization: Bearer $GAMMA_API_KEY" \
  https://api.gamma.app/v1/ping
```

### Step 2: Review Key Metrics

```bash
set -euo pipefail
# Check error rate (Prometheus)
curl -s 'http://prometheus:9090/api/v1/query?query=rate(gamma_requests_total{status=~"5.."}[5m])' | jq '.data.result'  # 9090: Prometheus port

# Check latency P95
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,rate(gamma_request_duration_seconds_bucket[5m]))' | jq '.data.result'  # Prometheus port

# Check rate limit
curl -s 'http://prometheus:9090/api/v1/query?query=gamma_rate_limit_remaining' | jq '.data.result'  # Prometheus port
```

### Step 3: Review Recent Logs

```bash
# Last 100 error logs
grep -i "gamma.*error" /var/log/app/gamma-*.log | tail -100

# Rate limit hits
grep "429" /var/log/app/gamma-*.log | wc -l  # HTTP 429 Too Many Requests

# Timeout errors
grep -i "timeout" /var/log/app/gamma-*.log | tail -50
```

## Incident Response Procedures

### Scenario 1: API Returning 5xx Errors

**Symptoms:**

- High error rate in monitoring
- Users reporting failed presentations
- 500/502/503 responses from Gamma

**Actions:**

1. Verify Gamma status: https://status.gamma.app
2. If Gamma outage confirmed:
   - Enable degraded mode / show maintenance message
   - Monitor status page for updates
   - No action needed on our side

3. If Gamma is operational:

   ```bash
   # Check our request patterns
   grep "5[0-9][0-9]" /var/log/app/gamma-*.log | \
     awk '{print $1}' | sort | uniq -c | sort -rn

   # Look for malformed requests
   grep -B5 "500" /var/log/app/gamma-*.log | grep "request"
   ```

4. Rollback recent deployments if issue correlates

### Scenario 2: Rate Limit Exceeded (429)

**Symptoms:**

- 429 responses in logs
- Rate limit metrics at zero
- Slow or queued requests

**Actions:**

1. Immediate mitigation:

   ```bash
   # Enable request throttling
   curl -X POST http://localhost:8080/admin/throttle \
     -d '{"gamma": {"rps": 10}}'
   ```

2. Check for runaway processes:

   ```bash
   # Find high-volume clients
   grep "gamma" /var/log/app/*.log | \
     awk '{print $5}' | sort | uniq -c | sort -rn | head -20
   ```

3. Enable circuit breaker:

   ```bash
   curl -X POST http://localhost:8080/admin/circuit-breaker \
     -d '{"service": "gamma", "state": "open"}'
   ```

4. Long-term: Review rate limit tier with Gamma

### Scenario 3: High Latency

**Symptoms:**

- Slow presentation creation
- Timeouts in logs
- P95 latency > 10s

**Actions:**

1. Check Gamma latency vs our latency:

   ```bash
   # Direct Gamma latency
   for i in {1..5}; do
     curl -w "%{time_total}\n" -o /dev/null -s \
       -H "Authorization: Bearer $GAMMA_API_KEY" \
       https://api.gamma.app/v1/ping
   done
   ```

2. If Gamma is slow:
   - Increase timeouts temporarily
   - Enable async mode for non-critical operations
   - Queue heavy operations

3. If our infrastructure is slow:
   - Check CPU/memory on app servers
   - Review connection pool settings
   - Check network connectivity

### Scenario 4: Authentication Failures (401/403)

**Symptoms:**

- All requests failing with 401
- "Invalid API key" errors
- Sudden authentication failures

**Actions:**

1. Verify API key:

   ```bash
   # Test key directly
   curl -H "Authorization: Bearer $GAMMA_API_KEY" \
     https://api.gamma.app/v1/ping

   # Check key format
   echo $GAMMA_API_KEY | head -c 20
   ```

2. If key is invalid:
   - Check if key was rotated
   - Deploy backup key: `GAMMA_API_KEY_SECONDARY`
   - Generate new key in Gamma dashboard

3. Notify team and update secrets

## Communication Templates

### Internal Notification

```
INCIDENT: Gamma Integration Issue

Severity: P

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.