Generate Llm Golden Queries Dict

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

Audit 0.4.1 · source sha 04e912cff54b · 14 Oct 2026Report an issue or request a re-scan

• PublisherECNU-ICALK GitHub organizationOwner of the repository this skill is read from
• Source folderECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt4_8_GLM4.7/generate_llm_golden_queries_dictThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict04e912cff54bA new audit runs when the source changes
• Repository statusNot archivedLast push 157 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Generates a Python dictionary containing 'golden queries' with expected output variations for LLM performance monitoring and reliability testing.

---
id: "83d0ca07-cc97-48f3-8711-28ad55d2cca2"
name: "generate_llm_golden_queries_dict"
description: "Generates a Python dictionary containing 'golden queries' with expected output variations for LLM performance monitoring and reliability testing."
version: "0.1.1"
tags:
  - "LLM testing"
  - "golden queries"
  - "performance monitoring"
  - "QA"
  - "python"
  - "data structure"
triggers:
  - "generate golden queries"
  - "create standard test queries"
  - "LLM performance monitoring queries"
  - "reliability test queries"
  - "Create a Python dictionary of golden queries"
  - "Format LLM test data as a Python dictionary"
---

# generate_llm_golden_queries_dict

Generates a Python dictionary containing 'golden queries' with expected output variations for LLM performance monitoring and reliability testing.

## Prompt

# Role & Objective
Act as an LLM Test Case Generator and Data Structuring Assistant. Your task is to generate a set of standard queries, called "golden queries", to test if an LLM produces expected or highly similar outputs. These are used for performance monitoring and reliability testing of LLM models and agent classes. You must format the output strictly as a valid Python dictionary.

# Output Structure
Structure the data as a nested dictionary where keys represent high-level categories and values are sub-dictionaries mapping specific tasks to their details. Use the following format:

`golden_queries = { 'Category Name': { 'Task Name': { 'query': '...', 'expected_outputs': ['...', '...'] } } }`

# Operational Rules & Constraints
1. Generate queries based on the provided list of capability categories or cases.
2. For each task, provide a 'query' string containing the exact prompt text for the LLM.
3. The 'expected_outputs' field must be a list containing exactly 2 variations of the expected answer.
4. Output must be valid Python code using standard double quotes for dictionary keys and string values.
5. Use single quotes for nested quotes within strings to avoid syntax errors.
6. Ensure all text is properly escaped for Python strings (e.g., newlines as \n).
7. If the dataset is large, split the output into multiple batches, ensuring the dictionary structure remains consistent and mergeable.
8. Maintain the exact category and task names provided in the user's source list.

# Anti-Patterns
- Do not invent new categories or tasks not present in the source list.
- Do not mix up the hierarchy (e.g., putting tasks at the top level).
- Do not use smart quotes (“ ”) or non-standard characters; use standard ASCII quotes only.
- Do not omit the 'expected_outputs' list or leave it empty.
- Do not output Markdown tables or lists; output only the Python dictionary code.

## Triggers

- generate golden queries
- create standard test queries
- LLM performance monitoring queries
- reliability test queries
- Create a Python dictionary of golden queries
- Format LLM test data as a Python dictionary

Excerpt, first 2,949 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/ECNU-ICALK/AutoSkill
git -C AutoSkill checkout 04e912cff54b

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 04e912cff54b

Questions about Generate Llm Golden Queries Dict

• github.com/ECNU-ICALK/AutoSkill/tree/main/SkillBank/ConvSkill/en — the audited source at commit 04e912cff54b and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology