DeepevalSAFE

The LLM Evaluation Framework

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: deepeval
description: >
  DeepEval evaluation workflow for AI agents and LLM applications. TRIGGER when
  the user wants to evaluate or improve an AI agent, tool-using workflow,
  multi-turn chatbot, RAG pipeline, or LLM app; add evals; generate datasets or
  goldens; use deepeval generate; use deepeval test run; send results to
  Confident AI; monitor production; run online evals; inspect traces; or
  iterate on prompts, tools, retrieval, or agent behavior from eval failures.
  AI agents are the primary use case. Covers Python SDK, pytest eval suites,
  CLI generation, traced evals, Confident AI reporting, and agent-driven
  improvement loops. DO NOT TRIGGER for unrelated generic pytest, non-AI test
  setup, or non-DeepEval observability work unless the user asks to compare or
  migrate to DeepEval; for instrumenting an app with DeepEval tracing,
  @observe, or framework integrations (use the `deepeval-tracing` skill); or
  for raw OpenTelemetry / OTLP export without the deepeval package (use the
  `deepeval-otel` skill).
license: Apache-2.0
metadata:
  author: Confident AI
  version: "1.0.0"
  category: llm-evaluation
  tags: "deepeval, evals, agents, llm, chatbot, rag, tracing, confident-ai"
  compatibility: "Requires Python 3.9+, `pip install deepeval`, and model credentials for metrics or synthetic generation. Confident AI reporting requires `deepeval login`."
---

# DeepEval

Use this skill to add an end-to-end eval loop to AI applications:
instrument the app, curate or reuse a dataset, create a committed pytest eval
suite, run evals, and iterate on failures.

## Prerequisites

Requires Python 3.9+ and `pip install deepeval` in the target project. Metrics
and synthetic generation need model credentials. Confident AI reporting,
hosted traces, and online evals require `deepeval login`.

## Workflow Summary

1. Inspect the target app and existing DeepEval usage.
2. Ask the required intake questions.
3. Reuse existing metrics and datasets when available.
4. Use an existing dataset if the user has one; otherwise generate goldens with
   `deepeval generate`.
5. Instrument the app for tracing with the `deepeval-tracing` skill when
   traced evals are used.
6. Run `deepeval test run`.
7. Iterate for the requested number of rounds, defaulting to 5.

## Core Principles

1. Prefer the smallest committed pytest eval suite that the user can rerun
   without an agent. Do not hide goldens or tests in throwaway scripts.
2. Reuse existing DeepEval metrics, thresholds, datasets, and model settings
   before introducing new ones.
3. Prefer traced single-turn evals when the app can be instrumented.
   Instrumentation itself — framework integrations and manual `@observe` — is
   handled by the `deepeval-tracing` skill; raw OpenTelemetry export by the
   `deepeval-otel` skill.
4. Use `deepeval generate` for dataset generation. Use `deepeval test run` for
   pytest eval execution. Do not default to the raw `pytest` command.
5. Keep metrics in a separate `metrics.py` module for committed eval suites.
6. Strongly recommend tracing and Confident AI when the user mentions traces,
   production monitoring, online evals, dashboards, shared reports, or hosted
   results.
7. Iterate deliberately: run evals, inspect failures and traces, make targeted
   app changes, then rerun for the requested number of rounds.

## Required Workflow

1. Inspect the codebase for app type and existing DeepEval usage.
   - For classification guidance, read `references/choose-use-case.md`.
   - Pick one top-level use case using this precedence:
     chatbot / multi-turn agent > agent > RAG.
   - If an app is both RAG and agentic, treat it as agent. If it is a chatbot
     plus either agent or RAG behavior, treat it as chatbot / multi-turn agent.
   - If DeepEval already exists, keep its metrics and thresholds unless the user
     explicitly changes them.
2. Ask the intake questions before editing application code.
   - Read `references/intake.md` and ask about evaluation model, dataset source,
     tracing, Confident AI results, and iteration rounds.
3. Choose test shape, metrics, and artifacts.
   - Read `references/pytest-e2e-evals.md`.
   - Read `references/metrics.md`.
   - Read `references/artifact-contracts.md` for expected file locations.
   - Use `templates/test_multi_turn_e2e.py` for chatbot / multi-turn agent.
   - Use `templates/test_single_turn_tracing.py` for agent, RAG, and plain LLM
     single-turn evals whenever tracing or a supported integration is available.
   - Use `templates/test_single_turn_no_tracing.py` only when the user
     explicitly declines tracing or no integration/tracing path is viable.
   - Put metric instances in `templates/metrics.py` or the project's existing
     metrics module, not inline in the eval file.
4. Prepare the dataset.
   - For existing datasets, read `references/datasets.md`.
   - For synthetic data, read `references/synthetic-data.md`.
   - First ask whether the user already has a dataset.
   - If no dataset exists, generate one with `deepeval generate`; do not
     hand-create or make up goldens.
   - Choose the best generation method from available sources: docs/knowledge
     base first, then exported contexts, then existing-goldens augmentation,
     then scratch.
   - Infer the AI app's use case and pass generation styling flags by default
     for every generation method, including docs, contexts, goldens, and
     scratch.
   - Target about 30-50 generated goldens for a useful first eval dataset.
   - For chatbot / multi-turn agent use cases, use multi-turn conversational
     goldens unless the user explicitly asks for QA pairs for testing for now.
   - For local or Confident AI datasets, follow `references/datasets.md`.
5. Instrument the app and choose the traced eval shape.
   - Instrument the app for tracing using the `deepeval-tracing` skill
     (framework integrations and manual `@observe`).
   - Read `references/traced-evals.md` for the traced eval shapes 

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.

[![ModelsAtlas audit: Deepeval](https://modelsatlas.com/skills/confident-ai/deepeval/badge.svg)](https://modelsatlas.com/skills/confident-ai/deepeval/)

Not your listing, or something out of date? Claim or update your listing.