eval-engineering

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

packages, open secret stores, or use credentials.

open `.env` files, credential stores, tokens, cookies, private keys, or secret

Audit 0.4.1 · source sha a78b2e2257e3 · 26 Oct 2026Report an issue or request a re-scan

• Publisherlangchain-ai GitHub organizationOwner of the repository this skill is read from
• Source folderlangchain-ai/langchain-skills/config/skills/eval-engineeringThe exact folder we audited
• LicenceMITAs the repository states it
• Commit pinned for this verdicta78b2e2257e3A new audit runs when the source changes
• Repository statusNot archivedLast push 1 day ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design,…

---
name: eval-engineering
description: Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.
---

# Eval Engineering

## Flow

- Inspect all inputs first: the repository, Harness, optional traces, existing
  Tasks and runs, existing World knowledge, and the human goal. Identify the
  source files and skill references that apply before proposing work.
- Create or update the small project World Knowledge Skill from reusable facts
  in those inputs. Use it to propose one grounded Task.
- Draft the Task Spec and World Skill together. Show both to the user, keep exact
  Task truth only in `Task.md`, and refine both until the user approves them.
- Implement the approved Task, validate its Environment and Verifier, run the
  real Harness, inspect the full evidence, and fix only non-agent failures.
- Reconcile the World Skill with what the run proved, then repeat this flow for
  the next Task.

## Terms

- **Task Spec:** `Task.md`, which describes the input, relevant agent
  conditions, Environment, scoring, fairness, and open decisions for one Task.
- **Task:** the runnable instruction, Environment, and Verifier.
- **Harness:** the complete agent Harbor runs, including prompts, model loop,
  tools, hooks, memory, sessions, and adapter.
- **Environment:** the files, data, services, identity, permissions, network,
  clock, and mutable state around the Harness.
- **Verifier:** independent checks that score the result or mark a run invalid.
- **World Knowledge Skill:** a repository-local skill with reusable
  project-specific knowledge, references, scripts, assets, and tests that help
  generate Task Specs and build future Tasks.
- **Spec2Task:** the full loop that turns a reviewed Task Spec into an audited
  runnable Task. Follow [Task implementation](references/task-implementation.md)
  for its concrete build order and reference routing.

## Reference routing

Read each reference when its decision appears:

| Need | Read |
|---|---|
| Inspect source, traces, the Harness, dependencies, access, and existing evals | [Discovery](references/discovery.md) |
| Bootstrap or update reusable project knowledge | [World knowledge](references/world-knowledge.md) |
| Propose Tasks and write the single Task Spec | [Task design](references/task-design.md) |
| Build data, services, access, state, and reset | [Environment building](references/environment-building.md) |
| Create structured or natural-language data | [Synthetic data](references/synthetic-data.md) |
| Define independent evidence and scoring | [Verifier design](references/verifier-design.md) |
| Apply Spec2Task to turn a reviewed Spec into an audited Task | [Task implementation](references/task-implementation.md) |
| Compare model runs and classify failures | [Calibration](references/calibration.md) |
| Package and run Harbor tasks | [Harbor](references/harbor.md) |
| Adapt a known benchmark design | [Benchmark patterns](references/patterns.md) |
| Build multi-turn conversations | [Multi-turn simulation](references/multi-turn-simulation/guide.md) |
| See World knowledge learned across two Tasks | [Service-desk example](references/examples/service-desk.md) |

Reusable implementation resources:

- Multi-turn [runner](references/multi-turn-simulation/runner.py), [model user](references/multi-turn-simulation/model_user.py), and [Harbor adapter example](references/multi-turn-simulation/harbor_example.py)
- [Tool-schema comparison](scripts/compare_tool_schemas.py), which compares
  supplied schema fragments but does not resolve external `$ref` targets
- [Read-only SQLite state snapshot](scripts/snapshot_sqlite_state.py)

## 1. Inspect inputs and existing World knowledge

Review every input the user provides before proposing a Task. Use the guidance
that matches each available input:

- For a repository, Harness, traces, or dependencies, read
  [Discovery](references/discovery.md).
- For existing Tasks and runs, inspect their instructions, Environments,
  Verifiers, rewards, trajectories, and final state. Read
  [Calibration](references/calibration.md) when run quality or failure causes
  affect the new design.
- For an existing project World Skill, read
  [World knowledge](references/world-knowledge.md), then check the sources and
  reusable methods that affect the new Task.
- For human goals and constraints, read
  [Task design](references/task-design.md).
- For relevant benchmark examples, use the domain index and source callouts in
  [Benchmark patterns](references/patterns.md).

Inspect the repository before asking questions that source and tests can
answer. Follow the active Harness through prompts, models, tools, services,
state, effects, and focused tests. Inspect existing Task instructions, parsers,
Verifiers, reward paths, and run evidence.

If the user supplies traces, review complete runs or threads. Use traces to
learn real requests, dependency behavior, state shapes, errors, and failure
conditions. Do not treat a trace answer as independent truth.

If `.agents/skills/<project>-world/SKILL.md` exists, read it. Follow its routing
only for knowledge relevant to the current Task. Check cited repository paths,
commands, and scripts when their accuracy affects the design.

## 2. Propose and select a Task

Read [Task design](references/task-design.md) and use the index in
[Benchmark patterns](references/patterns.md) to find the relevant domain and
source callouts. Focus on that domain unless the Task crosses another one.
In the first user-facing design response after inspection, propose one Task
grounded in repository evidence, supplied traces, existing coverage, or a
human priority. State:

- the real work and capability;
- the condition that makes the case

Excerpt, first 6,000 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

1. 2026-10-26
   ✓SAFE B 89 / 100

   first audit · source a78b2e2257e3

Questions about eval-engineering

• github.com/langchain-ai/langchain-skills/tree/main/config/skills — the audited source at commit a78b2e2257e3 and every finding aboveObserved · 26 Oct 2026

Spotted something wrong? Report a correction · Methodology