Groq Performance TuningSAFE

Model-agnostic agent-skills platform with a harness-free canonical layer, verified adapters, and the ccpi package manager. Explore at tonsofskills.com.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: groq-performance-tuning
description: 'Optimize Groq API performance with model selection, caching, streaming,
  and parallel requests.

  Use when experiencing slow responses, implementing caching strategies,

  or optimizing request throughput for Groq integrations.

  Trigger with phrases like "groq performance", "optimize groq",

  "groq latency", "groq caching", "groq slow", "groq speed".

  '
allowed-tools: Read, Write, Edit
version: 1.11.0
license: MIT
author: Jeremy Longshore <[email protected]>
tags:
- saas
- groq
- api
- performance
compatibility: Designed for Claude Code
---
# Groq Performance Tuning

## Overview

Maximize Groq's LPU inference speed advantage. Groq already delivers extreme throughput (280-560 tok/s) and low latency (<200ms TTFT), but client-side optimization -- model selection, prompt size, streaming, caching, and parallelism -- determines whether your application fully exploits that speed.

This skill walks through six tuning levers at a high level; the complete, copy-pasteable code for each lives in [references/implementation.md](references/implementation.md), and end-to-end worked scenarios live in [references/examples.md](references/examples.md).

## Prerequisites

- **Groq API key** — set `GROQ_API_KEY` in the environment. The `groq-sdk` client (`new Groq()`) reads it automatically; never hardcode the key.
- **Node.js 18+** with the `groq-sdk` package installed (`npm install groq-sdk`).
- Optional packages for the caching and parallelism steps: `lru-cache` and `p-queue` (`npm install lru-cache p-queue`).
- A baseline latency measurement of your current integration so you can confirm the tuning actually helps.

## Groq Speed Benchmarks

| Model | TTFT | Throughput | Context |
|-------|------|-----------|---------|
| `llama-3.1-8b-instant` | ~50ms | ~560 tok/s | 128K |
| `llama-3.3-70b-versatile` | ~150ms | ~280 tok/s | 128K |
| `llama-3.3-70b-specdec` | ~100ms | ~400 tok/s | 128K |
| `meta-llama/llama-4-scout-17b-16e-instruct` | ~80ms | ~460 tok/s | 128K |

TTFT = Time to First Token. Actual values depend on prompt size and server load.

## Instructions

Apply these six levers in order. Each is a small, independent change — start with the ones that match your bottleneck (model choice and caching give the biggest wins on most workloads). The full code for every step is in [references/implementation.md](references/implementation.md).

1. **Choose the right model for speed.** Map each call site to a speed tier: `llama-3.1-8b-instant` for latency-critical paths, `llama-3.3-70b-versatile` for quality-sensitive paths, `llama-3.3-70b-specdec` for 70b quality at higher throughput. Set `temperature: 0` so responses are deterministic (and cacheable).
2. **Minimize token count.** Trim verbose system prompts to their essence and set `max_tokens` to the expected output size, not a safe-looking ceiling. Fewer tokens means faster responses and less TPM-quota pressure.
3. **Stream for perceived performance.** For any output the user watches arrive, stream chunks and surface live TTFT / tokens-per-second metrics. Streaming hides TTFT even when total wall-clock is unchanged.
4. **Cache deterministic responses.** Hash `{messages, model}` and serve repeat `temperature: 0` requests from an LRU cache with a short TTL — turning a repeated call into a ~0ms hit.
5. **Parallelize under a rate-limit-aware queue.** Fan out bulk work with `p-queue`, capping concurrency and per-minute volume so you saturate throughput without tripping 429s.
6. **Benchmark before you commit.** Measure the candidate models against your real prompt shape and pick the fastest that clears your quality bar.

The essential skeleton — a tiered client every other step builds on:

```typescript
import Groq from "groq-sdk";

const groq = new Groq();  // reads GROQ_API_KEY from the environment

const SPEED_MAP = {
  instant: "llama-3.1-8b-instant",      // <100ms TTFT — latency-critical
  balanced: "llama-3.3-70b-versatile",  // <200ms TTFT — quality-sensitive
  fast70b: "llama-3.3-70b-specdec",     // 70b quality, faster throughput
} as const;

async function tieredCompletion(prompt: string, tier: keyof typeof SPEED_MAP = "instant") {
  return groq.chat.completions.create({
    model: SPEED_MAP[tier],
    messages: [{ role: "user", content: prompt }],
    temperature: 0,   // deterministic = cacheable
    max_tokens: 256,  // request only what you need
  });
}
```

See [references/implementation.md](references/implementation.md) for the streaming, caching, parallel-queue, and benchmarking functions in full.

## Output

Applying these levers to a Groq integration produces:

- **A tiered model map** (`SPEED_MAP`) so each call site uses the fastest model that meets its quality bar.
- **A streaming helper** that returns `{ content, ttftMs, totalMs, tokPerSec }` for live latency instrumentation.
- **A deterministic prompt cache** (LRU + SHA-256 key) that collapses repeated requests to ~0ms.
- **A rate-limit-aware parallel executor** that maximizes throughput without hitting 429s.
- **A benchmark report** printing average latency and tokens/sec per model, e.g.:

```text
llama-3.1-8b-instant     |  61ms avg | 548 tok/s avg
llama-3.3-70b-versatile  | 148ms avg | 279 tok/s avg
llama-3.3-70b-specdec    | 103ms avg | 401 tok/s avg
```

## Performance Decision Matrix

| Scenario | Model | max_tokens | stream | cache |
|----------|-------|-----------|--------|-------|
| Classification | 8b-instant | 5 | No | Yes |
| Chat response | 70b-versatile | 1024 | Yes | No |
| Data extraction | 8b-instant | 200 | No | Yes |
| Code generation | 70b-versatile | 2048 | Yes | No |
| Bulk processing | 8b-instant | 256 | No | Yes |

## Examples

Common scenarios mapped to the levers above. Full code for each is in [references/examples.md](references/examples.md).

- **Latency-critical classification** — `8b-instant` + one-word prompt + `max_tokens: 5` + cache. First call ~50ms TTFT; identical repeats return from cache at ~0m

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.