Scholar EvaluationSAFE

Kosmos: An AI Scientist for Autonomous Discovery - An implementation and adaptation to be driven by Claude Code or API - Based on the Kosmos AI Paper - https://arxiv.org/abs/2511.02824

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: scholar-evaluation
description: Systematic framework for evaluating scholarly and research work based on the ScholarEval methodology. This skill should be used when assessing research papers, evaluating literature reviews, scoring research methodologies, analyzing scientific writing quality, or applying structured evaluation criteria to academic work. Provides comprehensive assessment across multiple dimensions including problem formulation, literature review, methodology, data collection, analysis, results interpretation, and scholarly writing quality.
---

# Scholar Evaluation

## Overview

Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.

## When to Use This Skill

Use this skill when:
- Evaluating research papers for quality and rigor
- Assessing literature review comprehensiveness and quality
- Reviewing research methodology design
- Scoring data analysis approaches
- Evaluating scholarly writing and presentation
- Providing structured feedback on academic work
- Benchmarking research quality against established criteria

## Evaluation Workflow

### Step 1: Initial Assessment and Scope Definition

Begin by identifying the type of scholarly work being evaluated and the evaluation scope:

**Work Types:**
- Full research paper (empirical, theoretical, or review)
- Research proposal or protocol
- Literature review (systematic, narrative, or scoping)
- Thesis or dissertation chapter
- Conference abstract or short paper

**Evaluation Scope:**
- Comprehensive (all dimensions)
- Targeted (specific aspects like methodology or writing)
- Comparative (benchmarking against other work)

Ask the user to clarify if the scope is ambiguous.

### Step 2: Dimension-Based Evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to `references/evaluation_framework.md` for detailed criteria and rubrics for each dimension.

**Core Evaluation Dimensions:**

1. **Problem Formulation & Research Questions**
   - Clarity and specificity of research questions
   - Theoretical or practical significance
   - Feasibility and scope appropriateness
   - Novelty and contribution potential

2. **Literature Review**
   - Comprehensiveness of coverage
   - Critical synthesis vs. mere summarization
   - Identification of research gaps
   - Currency and relevance of sources
   - Proper contextualization

3. **Methodology & Research Design**
   - Appropriateness for research questions
   - Rigor and validity
   - Reproducibility and transparency
   - Ethical considerations
   - Limitations acknowledgment

4. **Data Collection & Sources**
   - Quality and appropriateness of data
   - Sample size and representativeness
   - Data collection procedures
   - Source credibility and reliability

5. **Analysis & Interpretation**
   - Appropriateness of analytical methods
   - Rigor of analysis
   - Logical coherence
   - Alternative explanations considered
   - Results-claims alignment

6. **Results & Findings**
   - Clarity of presentation
   - Statistical or qualitative rigor
   - Visualization quality
   - Interpretation accuracy
   - Implications discussion

7. **Scholarly Writing & Presentation**
   - Clarity and organization
   - Academic tone and style
   - Grammar and mechanics
   - Logical flow
   - Accessibility to target audience

8. **Citations & References**
   - Citation completeness
   - Source quality and appropriateness
   - Citation accuracy
   - Balance of perspectives
   - Adherence to citation standards

### Step 3: Scoring and Rating

For each evaluated dimension, provide:

**Qualitative Assessment:**
- Key strengths (2-3 specific points)
- Areas for improvement (2-3 specific points)
- Critical issues (if any)

**Quantitative Scoring (Optional):**
Use a 5-point scale where applicable:
- 5: Excellent - Exemplary quality, publishable in top venues
- 4: Good - Strong quality with minor improvements needed
- 3: Adequate - Acceptable quality with notable areas for improvement
- 2: Needs Improvement - Significant revisions required
- 1: Poor - Fundamental issues requiring major revision

To calculate aggregate scores programmatically, use `scripts/calculate_scores.py`.

### Step 4: Synthesize Overall Assessment

Provide an integrated evaluation summary:

1. **Overall Quality Assessment** - Holistic judgment of the work's scholarly merit
2. **Major Strengths** - 3-5 key strengths across dimensions
3. **Critical Weaknesses** - 3-5 primary areas requiring attention
4. **Priority Recommendations** - Ranked list of improvements by impact
5. **Publication Readiness** (if applicable) - Assessment of suitability for target venues

### Step 5: Provide Actionable Feedback

Transform evaluation findings into constructive, actionable feedback:

**Feedback Structure:**
- **Specific** - Reference exact sections, paragraphs, or page numbers
- **Actionable** - Provide concrete suggestions for improvement
- **Prioritized** - Rank recommendations by importance and feasibility
- **Balanced** - Acknowledge strengths while addressing weaknesses
- **Evidence-based** - Ground feedback in evaluation criteria

**Feedback Format Options:**
- Structured report with dimension-by-dimension analysis
- Annotated comments mapped to specific document sections
- Executive summary with key findings and recommendations
- Comparative analysis against benchmark standards

### Step 6: Contextual Considerations

Adjust evaluation approach based on:

**Stage of Development:**
- Early draft: Focus on conceptual and structural issues
- Advanced draft: Focus on refinement and polish
- Final submission: Comprehe

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.