Media Processing

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

assistant/CLAUDE.md

assistant/src/approvals/CLAUDE.md

assistant/src/notifications/CLAUDE.md

assistant/src/runtime/CLAUDE.md

gateway/CLAUDE.md

import type { BatchTranscriber } from "../../../../stt/types.js";

mock.module("../../../../providers/speech-to-text/resolve.js", () => ({

mock.module("../../../../util/spawn.js", () => ({

import { createAbortReason } from "../../../../util/abort-reasons.js";

mock.module("../../../../persistence/media-store.js", () => ({

Audit 0.4.1 · source sha 19bc86343b63 · 14 Oct 2026Report an issue or request a re-scan

• Publishervellum-ai GitHub organizationOwner of the repository this skill is read from
• Source foldervellum-ai/vellum-assistant/assistant/src/config/bundled-skills/media-processingThe exact folder we audited
• LicenceMITAs the repository states it
• Commit pinned for this verdict19bc86343b63A new audit runs when the source changes
• Repository statusNot archivedLast push 1 day ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Ingest and process media files (video, audio, image)

---
name: media-processing
description: "Ingest and process media files (video, audio, image)"
compatibility: "Designed for Vellum personal assistants"
metadata:
  emoji: "🎬"
  vellum:
    display-name: "Media Processing"
    category: "content"
    activation-hints:
      - "User wants to ingest, process, or analyze a local video, audio, or image file"
      - "User wants to ask natural-language questions about a video's content (e.g. 'what happens at 3:14', 'find all the goals', 'who appears in this footage')"
      - "User wants to extract a clip or short segment from a video"
      - "User wants to run keyframe extraction, segmentation, or frame-by-frame analysis on a video"
    avoid-when:
      - "User only wants a plain audio transcript — use the transcribe skill instead"
      - "User just wants to play or open a media file in their default system player — that does not need this pipeline"
---

Ingest and track processing of media files (video, audio, images) through a configurable 3-phase pipeline.

## End-to-End Workflow

The processing pipeline follows a sequential 3-phase flow:

1. **Ingest** (`ingest_media`) - Register a media file, detect MIME type, extract duration, deduplicate by content hash.
2. **Preprocess** (`extract_keyframes`) - Detect dead time, segment the video into windows, extract downscaled keyframes, build a subject registry, and write a pipeline manifest.
3. **Map** (`analyze_keyframes`) - Send each segment's frames to the configured Gemini vision model with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability, and automatic retries.
4. **Reduce / Query** (`query_media`) - Send all map output to Claude for intelligent analysis and Q&A. Supports arbitrary natural language queries about video content.
5. **Clip** (`generate_clip`) - Extract video clips around specific moments.

The processing pipeline service (`services/processing-pipeline.ts`) orchestrates phases 2-4 automatically with retries, resumability, and cancellation support.

## Tools

### ingest_media

Register a media file for processing. Accepts an absolute file path, validates the file exists, detects MIME type, extracts duration (for video/audio via ffprobe), and registers the asset with content-hash deduplication.

### media_status

Query the processing status of a media asset. Returns the asset metadata along with per-stage progress details. Use this to monitor pipeline progress.

### extract_keyframes

Preprocess a video asset: detect dead time via mpdecimate, segment the video into windows, extract downscaled keyframes at regular intervals, build a subject registry, and write a pipeline manifest.

Parameters:

- `asset_id` (required) - ID of the media asset.
- `interval_seconds` - Interval between keyframes (default: 1s). Use 0.5s for sports/action content where frame density matters.
- `segment_duration` - Duration of each segment window (default: 15s).
- `dead_time_threshold` - Sensitivity for dead-time detection (default: 0.02).
- `section_config` - Path to a JSON file with manual section boundaries.
- `detect_dead_time` - Whether to detect and skip dead time (default: false). Dead-time detection can be too aggressive for continuous action video like sports - it may incorrectly skip live play. Enable only for content with clear idle periods (e.g., lectures, surveillance footage).
- `short_edge` - Short edge resolution for downscaled frames in pixels (default: 480).
- `include_audio` - Whether to extract and transcribe audio for each segment (default: false). When enabled, each segment's audio is transcribed using the configured STT service and stored alongside visual frames.

### analyze_keyframes

Map video segments through Gemini's structured output API. Supports two modes:

- **`keyframes`** (default) - Reads frames from the preprocess manifest, sends each segment's images to Gemini. Requires `extract_keyframes` to be run first. Best for longer videos (> 1 hour) or when you need fine-grained control over frame selection (interval, segment duration, dead-time skipping).
- **`direct_video`** - Uploads the video file directly to Gemini's Files API. Gemini sees actual motion and temporal context instead of static frames. Best for shorter videos (< 1 hour) where temporal context matters (detecting actions, transitions, motion patterns). Has a 2 GB file size limit. Does not require `extract_keyframes` preprocessing.

Both modes produce the same `MapOutput` format, so `query_media` works identically regardless of which mode was used.

Parameters:

- `asset_id` (required) - ID of the media asset.
- `system_prompt` (required) - Extraction instructions for Gemini.
- `output_schema` (required) - JSON Schema for structured output.
- `mode` - Analysis mode: `'keyframes'` (default) or `'direct_video'`.
- `context` - Additional context to include in the prompt.
- `model` - Gemini model to use (defaults to the Gemini provider's recommended vision model).
- `concurrency` - Maximum concurrent API requests (default: 10, keyframes mode only).
- `max_retries` - Retry attempts per segment on failure (default: 3).

### query_media

Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content.

Parameters:

- `asset_id` (required) - ID of the media asset.
- `query` (required) - Natural language query about the video data.
- `system_prompt` - Optional system prompt for Claude.
- `model` - LLM model to use. Defaults to the assistant's active chat model.

### generate_clip

Extract a video clip from a media asset using ffmpeg. Applies configurable pre/post-roll padding (clamped to file boundaries), outputs the clip as a temporary file.

## Services

### Processing Pipeline (services/processing-pipeline.ts)

Orchestrates the full processing pipeline with reliability features:

- **Sequential e

Excerpt, first 6,001 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/vellum-ai/vellum-assistant
git -C vellum-assistant checkout 19bc86343b63

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 19bc86343b63

Questions about Media Processing

• github.com/vellum-ai/vellum-assistant/tree/main/assistant/src/co — the audited source at commit 19bc86343b63 and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology