Live survey304 models · 14 providers · 23,592 auditsAudited implement · SAFE →What changedHow we verifyllms.txt

Agent skill · Audited from source

trl-training

skill · huggingface/skillsApache-2.0 licenceAudited 14 Nov 2026 · source 33087d8174de

★ 11,343pushed today
✓SAFEGrade B · 89/100 · 14 Nov 2026 · 33087d8
11,343repo stars · whole huggingface/skills

Is it safe? Yes — SAFE, grade B (89/100). We read this skill's source at commit 33087d8174de and found no filesystem, network or shell access and no hard-coded secrets. What it does: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.

Ask an AI about this pageGChatGPTCClaudePPerplexityGeGeminiXGrok
More
Cite

Cite this page

ModelsAtlas. “trl-training (huggingface/skills)”. Retrieved 14 Nov 2026. https://modelsatlas.com/skills/huggingface/trl-training

Trust audit

Audit 0.4.1 · source read 14 Nov 2026

Trust score89/100✓SAFE B
0 critical0 high0 medium0 low0 info

SAFE means nothing in the source contradicts what the skill says it does. Grade B is shared by 17,406 of 18,389 audited skills.

What the source does

  • FilesystemNone observedNo reads or writes in the source
  • NetworkNone observedNo outbound calls in the source
  • ShellNone observedNo commands run in the source
  • DependenciesPinnedNothing floats
  • SecretsNone foundNo keys in source

Five audit layers

LayerWhat we checkResultFindings
L0 ProvenanceRepo owner, licence, archive status, activityPASS0
L1 Static sourceSecrets, path traversal, supply chain, hidden codeNA0
L2 InstructionsPrompt injection, authority framing, hidden directivesPASS0
L3 InventoryThe files and declarations match the sourcePASS0
L4 SandboxRun in isolation and watch behaviourSKIPPED why0

Gates applied: no behavioural pass.

Findings (0)

No findings at any severity. We re-audit when the source changes.

Audit 0.4.1 · source sha 33087d8174de · 14 Nov 2026Report an issue or request a re-scan

Provenance

Who published it, and what the verdict is pinned to

  • Publisherhuggingface GitHub organizationOwner of the repository this skill is read from
  • Source folderhuggingface/skills/skills/trl-trainingThe exact folder we audited
  • LicenceApache-2.0As the repository states it
  • Commit pinned for this verdict33087d8174deA new audit runs when the source changes
  • Repository statusNot archivedPushed today
  • Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

What it tells the agent

Full source on GitHub ↗

Its description: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.

---
name: trl-training
description: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
license: Apache-2.0
metadata:
  author: huggingface
  documentation: https://huggingface.co/docs/trl
---

# TRL

Each method pairs a `*Trainer` class with a `*Config` dataclass. Configs extend `transformers.TrainingArguments`, so all of its arguments work in any trainer config.

| Trainer | Dataset type |
|---|---|
| `SFTTrainer` | language modeling or prompt-completion |
| `DPOTrainer` | preference (chosen/rejected pairs) |
| `GRPOTrainer` | prompt-only + reward function(s) |
| `DistillationTrainer` | prompt-only + a teacher model (on-policy distillation) |
| `KTOTrainer` | unpaired preference (per-sample bool label) |
| `RewardTrainer` | preference (chosen/rejected pairs); trains a scalar reward model, not a policy |

Many more trainers (OnlineDPO, ORPO, CPO, GKD, ...) live in `trl.experimental` with unstable APIs: https://huggingface.co/docs/trl/experimental_overview

```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",  # model ID or a PreTrainedModel instance
    args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()
```

Pass `model` as a string and route loading kwargs through `model_init_kwargs` (e.g. `{"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}`) instead of calling `from_pretrained` yourself. The tokenizer/processor is inferred from the model; pass `processing_class` only when it differs. For LoRA, pass `peft_config=LoraConfig(...)`.

## Dataset formats

Conversational: `{"messages": [{"role": ..., "content": ...}]}` (language modeling) or `{"prompt": [...], "completion": [...]}`. The chat template is applied automatically — never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats

## SFT: the fields that matter

```python
SFTConfig(
    max_length=1024,        # truncation length; None disables truncation
    packing=True,           # pack sequences into max_length blocks: fewer pad tokens, higher throughput
    padding_free=True,      # flatten batch, no padding; requires FlashAttention; implied by packing
    use_liger_kernel=True,  # fused Liger kernels, reduces peak memory
    assistant_only_loss=True,  # loss only on assistant turns (conversational datasets)
)
```

## GRPO: online RL

```python
def reward_len(completions, **kwargs):
    return [-abs(20 - len(c[0]["content"])) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,  # or a list; rewards are summed
    args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", max_completion_length=512),
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
```

Reward functions are called with keyword arguments `prompts`, `completions`, `completion_ids`, `trainer_state`, plus every extra dataset column — accept `**kwargs` for the ones you ignore. Return `list[float]`, one reward per completion. With conversational data, `completions` is a list of message lists, not strings.

The generation batch is `per_device_train_batch_size × num_processes × steps_per_generation` (or set `generation_batch_size` directly) and must be divisible by `num_generations` (default 8). Generation is the usual bottleneck — enable vLLM with `use_vllm=True`: `vllm_mode="colocate"` shares the training GPUs (size with `vllm_gpu_memory_utilization`); `vllm_mode="server"` uses a separate `trl vllm-serve --model <model_id>`.

`AsyncGRPOTrainer` (`trl.experimental.async_grpo`) implements the same algorithm with generation decoupled from training: a background worker streams completions from a vLLM server while the training loop consumes them, so the two overlap instead of alternating.

## CLI

Flags mirror the config fields: `trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara`. YAML via `--config`; distributed presets via `--accelerate_config zero3` (Python scripts: `accelerate launch train.py`).

Excerpt, first 4,312 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

Install

Pinned to the audited commit 33087d8

Download folder ↗
git clone https://github.com/huggingface/skills
git -C skills checkout 33087d8174de

The author documents no install command, so this uses the repository directly. Our verdict covers commit 33087d8174de only.

Works in

The author has not said which agents this skill supports, and we have not tested it in any.

Audit history

Every audit, every source change

  1. ✓SAFE B 89 / 100

    first audit · source 33087d8174de

README badge

For maintainers · shows the latest audit

ModelsAtlas: SAFE BMODELSATLASSAFE · B
[![ModelsAtlas: SAFE B](https://modelsatlas.com/skills/huggingface/trl-training/badge.svg)](https://modelsatlas.com/skills/huggingface/trl-training/)

The badge shows our verdict and grade for this skill, and links back to this page. It changes when the audit does.

Similar skills

Add this badge to your README

[![ModelsAtlas: SAFE B](https://modelsatlas.com/skills/huggingface/trl-training/badge.svg)](https://modelsatlas.com/skills/huggingface/trl-training/)

Questions about trl-training

What does the trl-training skill do?
Give your agents the power of the Hugging Face ecosystem
Is trl-training safe to install?
The audit found nothing in the source that contradicts what it says it does, and graded it B (89/100). Grade A is held back for packages that have also passed a sandboxed behavioural run, which is why a clean skill reads B.
What can trl-training access on my machine?
The audit observed no filesystem, network or shell use at all in its source.
How current is this page?
The grade is for one exact copy of the source (33087d8174de), read on 2026-11-14. The repository is watched, and a new audit runs when it changes — this is the first audit.

Sources

Spotted something wrong? Report a correction · Methodology

Keep going in your AI

Opens with this page as the source. We add nothing else to the prompt.