Sn Da Image CaptionSAFE

Modular SenseNova skills for building AI-powered office assistants and productivity workflows

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: sn-da-image-caption
description: "图片理解与数据提取 skill。当图片文件(.png/.jpg/.jpeg/.gif/.webp/.bmp)是主要输入且用户需要理解、提取数据或分析图片内容时使用。提供预配置的 caption 脚本(scripts/caption.py),通过 vision 模型将图片转为文本描述,无需额外配置 API Key。覆盖:(1) 通过 scripts/caption.py 对图表/表格/截图/流程图进行 caption,(2) 将 caption 文本解析为结构化 DataFrame,(3) 基于提取数据重新生成可视化图表,(4) 导出为 Excel/CSV。**遇到以下任一情况就主动使用本 skill,不要自行猜测图片内容**:1用户出现触发词:图片分析 / 图表提取 / 表格识别 / OCR / 图片描述 / 截图分析 / 图表数据 / 提取图片中的数据 / 图片转表格 / 识别图片 / image caption / extract data from image / chart analysis / table OCR;2用户上传或指定了图片文件(.png / .jpg / .jpeg / .gif / .webp / .bmp)并要求理解、提取数据或分析内容;3任务需要从图表截图、表格截图、UI 截图、流程图中提取结构化信息;4用户要求将图片中的数据转为 Excel/CSV 或重新生成可视化图表。仅不用于:图片编辑(裁剪、滤镜、缩放)、图片生成、不含数据的风景/人物照片描述。"
---

# Image Caption Analysis — 图片描述与数据提取

## Overview

Analyze, extract data from, or understand image files (.png, .jpg, .jpeg, .gif, .webp, .bmp). The core workflow:

1. Run `scripts/caption.py` to get a text description of the image
2. Parse the description into structured data (DataFrame, etc.)
3. Analyze, visualize, or export

## scripts/caption.py — Image Caption

The script converts images to text descriptions via a vision model. Configure via `SN_API_KEY` (minimum required), or use `SN_VISION_API_KEY` / `SN_VISION_BASE_URL` / `SN_VISION_MODEL` for fine-grained control. See the project environment variable spec for the full fallback chain.

### Usage

```bash
# Basic — get text description
python3 scripts/caption.py /mnt/data/image.png

# Custom prompt — guide what to extract
python3 scripts/caption.py /mnt/data/chart.png --prompt "提取所有数值,Markdown 表格格式"

# JSON output — includes detected type, usage stats, cache info
python3 scripts/caption.py /mnt/data/image.png --json

# Batch — process all images in a directory
python3 scripts/caption.py /mnt/data/images/ --batch --output /mnt/data/captions.json

# Override model (optional)
python3 scripts/caption.py /mnt/data/image.png --model gemini-3.1-flash-lite-preview
```

### Options

| Option | Description |
|--------|------------|
| `--prompt, -p` | Custom prompt (overrides auto-detection) |
| `--model, -m` | Vision model (default: sensenova-6.8-flash-lite) |
| `--json` | Output structured JSON instead of plain text |
| `--batch` | Process all images in a directory |
| `--output, -o` | Output file for batch results |
| `--no-cache` | Skip MD5 cache |

### What it does automatically

- **Type detection**: Detects image type from filename (chart/table/UI/diagram/general) and picks the best prompt
- **Compression**: Images >5MB or >2048px are compressed before sending
- **Caching**: Same image + same prompt → instant cached result, no API cost
- **Error handling**: Retries on failure, returns error message on permanent failure

### JSON output format

```json
{
  "file": "/mnt/data/image.png",
  "type": "chart",
  "description": "这是一张柱状图...",
  "usage": {"prompt_tokens": 1100, "completion_tokens": 400},
  "cached": false
}
```

## Calling from Python

```python
import subprocess, json

CAPTION = "/path/to/skills/sn-da-image-caption/scripts/caption.py"

# Single image
result = subprocess.run(
    ["python3", CAPTION, "/mnt/data/chart.png", "--json",
     "--prompt", "提取图表数据,Markdown 表格输出"],
    capture_output=True, text=True, timeout=60
)
data = json.loads(result.stdout)
description = data["description"]

# Batch
result = subprocess.run(
    ["python3", CAPTION, "/mnt/data/images/", "--batch",
     "--output", "/mnt/data/captions.json"],
    capture_output=True, text=True, timeout=300
)
with open("/mnt/data/captions.json") as f:
    all_captions = json.load(f)
```

## Prompt Strategy

Different image types need different prompts. The script auto-detects, but specifying `--prompt` gives better results.

| Image Type | When | Recommended --prompt |
|-----------|------|---------------------|
| Data chart | 柱状图/折线图/饼图 | `"提取图表标题、坐标轴、每个数据点数值、图例。Markdown 表格输出。"` |
| Table screenshot | 表格截图 | `"提取表格所有内容,Markdown 表格格式,保持行列结构,数值不四舍五入。"` |
| UI screenshot | 界面截图 | `"以前端开发者视角描述:布局、组件、文字、颜色。"` |
| Diagram | 流程图/架构图 | `"描述所有节点、连接关系(A→B)、分支条件。"` |
| General | 照片、其他 | 不传 --prompt,用默认 |

## Parsing Caption Results

Caption 通常返回 Markdown 表格,解析为 DataFrame:

```python
import pandas as pd

def parse_markdown_table(text):
    lines = text.strip().split('\n')
    table_lines = []
    in_table = False
    for line in lines:
        stripped = line.strip()
        if '|' in stripped:
            in_table = True
            table_lines.append(stripped)
        elif in_table:
            break

    data_lines = []
    for l in table_lines:
        cells = [c.strip() for c in l.split('|') if c.strip()]
        if cells and not all(set(c) <= set('-: ') for c in cells):
            data_lines.append(cells)

    if len(data_lines) < 2:
        return None

    header = data_lines[0]
    rows = [r for r in data_lines[1:] if len(r) == len(header)]
    df = pd.DataFrame(rows, columns=header)

    # Auto numeric conversion
    for col in df.columns:
        try:
            cleaned = df[col].str.replace(',', '').str.strip()
            if cleaned.str.endswith('%').any():
                df[col] = pd.to_numeric(cleaned.str.rstrip('%'), errors='coerce')
            else:
                converted = pd.to_numeric(cleaned, errors='coerce')
                if converted.notna().sum() > len(df) * 0.5:
                    df[col] = converted
        except Exception:
            pass
    return df
```

## Visualization

### Chinese Font Setup (MANDATORY)

```python
import matplotlib.pyplot as plt
import matplotlib
import os

font_path = '/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc'
if os.path.exists(font_path):
    matplotlib.rcParams['font.family'] = 'WenQuanYi Zen Hei'
matplotlib.rcParams['axes.unicode_minus'] = False
```

### Color Palette

```python
COLORS = ['#4C72B0', '#55A868', '#C44E52', '#8172B2', '#CCB974', '#64B5CD']
```

### Save & Display

```python
plt.savefig('/mnt/data/chart.png', dpi=150, bbox_inches='tight')
plt.show()
print("![图表](sandbox:/mnt/data/chart.png)")
```

## Export 

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (2)

return hashlib.md5(data, usedforsecurity=False)

return hashlib.md5(data)

Gates applied: no_behavioural_pass.