Llm Cost Optimizer

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

assistant/CLAUDE.md

assistant/src/approvals/CLAUDE.md

assistant/src/notifications/CLAUDE.md

assistant/src/runtime/CLAUDE.md

gateway/CLAUDE.md

Audit 0.4.1 · source sha 19bc86343b63 · 14 Oct 2026Report an issue or request a re-scan

• Publishervellum-ai GitHub organizationOwner of the repository this skill is read from
• Source foldervellum-ai/vellum-assistant/skills/llm-cost-optimizerThe exact folder we audited
• LicenceMITAs the repository states it
• Commit pinned for this verdict19bc86343b63A new audit runs when the source changes
• Repository statusNot archivedLast push 1 day ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Analyze and reduce LLM spend: read usage breakdowns by call site, model, and inference profile, understand single-winner profile resolution, and pin call sites to managed profiles (Balanced / Quality / Budget / Fast)…

---
name: "llm-cost-optimizer"
description: "Analyze and reduce LLM spend: read usage breakdowns by call site, model, and inference profile, understand single-winner profile resolution, and pin call sites to managed profiles (Balanced / Quality / Budget / Fast) only where they should deviate from shipped defaults."
metadata:
  emoji: "💸"
  vellum:
    category: "development"
    display-name: "LLM Cost Optimizer"
---

## Overview

This skill walks through analyzing and reducing LLM spend on a Vellum assistant. There are three layers:

1. **Provider connections** — named auth configs (e.g. `anthropic-managed`, `my-personal-key`)
2. **Model profiles** — named presets (provider + model + effort + thinking + contextWindow). Four managed defaults, with UI labels. Note that the keys do not track the labels: read the key, not the name, when pinning a call site.
   - `balanced` → **Balanced** (the general agent-loop profile)
   - `quality-optimized` → **Quality** (the expensive escalation profile)
   - `cost-optimized` → **Budget** (the cheap utility/background profile, and the one to pin for spend reduction)
   - `latency-optimized` → **Fast** (the low time-to-first-token profile, used by live voice; faster but not cheaper than Budget)
3. **Call-site profile pins** (`llm.callSites.<id>.profile`) — optional per-task overrides of the shipped defaults.

The concrete model behind each managed profile depends on the install: platform-managed installs and BYOK installs resolve different providers/models, and the catalog changes over time. **Never assume which model a profile maps to** — read `assistant config get llm.profiles` and the usage breakdown by `model` to see what actually ran.

## How model selection works — read this before diagnosing

Every LLM call resolves exactly **one winning profile** through a strict first-usable-wins chain. Profiles never merge with each other:

1. **Per-conversation / per-run override** — the user's `/model` pick, an open `assistant inference session`, or a schedule's pinned profile
2. **`llm.activeProfile`** — applies to `mainAgent` (the chat loop) **only**; it IS the user's chat-model selection and outranks any `llm.callSites.mainAgent` pin
3. **`llm.callSites.<site>.profile`** — explicit per-site pin
4. **The call site's shipped default intent**, resolved through `llm.defaultProvider`
5. `balanced` intent (final anchor)

A rung only wins if its profile exists, is enabled, and carries its own provider + model; otherwise resolution silently falls to the next rung.

Consequences that change how you diagnose cost:

- **A missing or empty `llm.callSites` block is healthy, not a red flag.** Every call site ships with a sensible default intent: the agent loop and quality-sensitive sites (`mainAgent`, `subagentSpawn`, `compactionAgent`, `callAgent`, `patternScan`, `narrativeRefinement`, `memoryConsolidation`, `memoryV2Consolidation`, `memoryV3SelectL2`, `recall`, `conversationStarters`, `identityIntro`, `emptyStateGreeting`) default to `balanced`; everything else (classifiers, summarization, titles, copy generation, memory extraction/retrieval/sweeps, heartbeat, home-screen content, etc.) defaults to `cost-optimized`. Nothing "falls back" to an expensive model.
- **Do not write a full `llm.callSites` blob that mirrors the shipped defaults.** That freezes today's defaults into user config and silently opts the user out of future default improvements (and of tuning shipped alongside them, like cache and context-window settings). Pin only deliberate deviations.

## Step 1 — Measure current spend

```bash
# Monthly totals
assistant usage totals --range month

# Break down by conversation (what the user actually did — use this for presentation)
assistant usage breakdown --group-by conversation --range month

# Break down by call site (what kind of work is expensive — use for diagnosis)
assistant usage breakdown --group-by call_site --range month

# Break down by model (what actually ran)
assistant usage breakdown --group-by model --range month

# Break down by profile (which selection produced it)
assistant usage breakdown --group-by inference_profile --range month
```

Cross-reference the `call_site` and `inference_profile` breakdowns: a background call site showing spend under an expensive profile means an override or pin routed it there — that is the interesting finding, not the config defaults.

Add `--json` when you need token-level detail (input vs output vs `cache_creation` vs `cache_read`) — high input volume on a cheap model can outweigh low volume on an expensive one.

## Step 1b — Present costs in user-friendly terms

After gathering the data, present findings in a format the user can act on. Users think in terms of conversations they had and automations they set up — not call sites, inference profiles, or cache economics. Use the `call_site`, `model`, and `inference_profile` breakdowns for **diagnosis**, but lead the presentation with what the user recognizes.

**Presentation structure:**

1. **One-line headline** with total monthly cost.
2. **Conversations table** — the user's own activity, sorted by cost descending. Columns: conversation name, turns, cost, and a brief note on why it was expensive (e.g. "One heavy session drove 65% of your total spend"). Roll up small conversations into an "All other conversations" row to keep the table to 4-6 rows.
3. **"What I'd change to cut costs"** — 2-3 bullets in plain English, biggest lever first. No jargon. Instead of "drop the balanced profile effort from high to medium," say "your chat model is set to high effort — dropping to medium would save ~\$X/week." Each bullet states what to change, why it helps, and the estimated savings.
4. **A clear ask** — "Want me to make either of those changes?"

**What to omit from the user-facing presentation:**

- **Background work** (memory processing, heartbeats, health checks) — users can't control these individually and they're already on cheap models by default. Mentioning them

Excerpt, first 6,001 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/vellum-ai/vellum-assistant
git -C vellum-assistant checkout 19bc86343b63

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 19bc86343b63

Questions about Llm Cost Optimizer

• github.com/vellum-ai/vellum-assistant/tree/main/skills/llm-cost- — the audited source at commit 19bc86343b63 and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology