Agent skill · Audited from source
trl-training
skill · huggingface/skillsApache-2.0 licenceAudited 14 Nov 2026 · source 33087d8174de
Is it safe? Yes — SAFE, grade B (89/100). We read this skill's source at commit 33087d8174de and found no filesystem, network or shell access and no hard-coded secrets. What it does: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
Cite
Cite this page
ModelsAtlas. “trl-training (huggingface/skills)”. Retrieved 14 Nov 2026. https://modelsatlas.com/skills/huggingface/trl-training
Trust audit
Audit 0.4.1 · source read 14 Nov 2026
SAFE means nothing in the source contradicts what the skill says it does. Grade B is shared by 17,406 of 18,389 audited skills.
What the source does
- FilesystemNone observedNo reads or writes in the source
- NetworkNone observedNo outbound calls in the source
- ShellNone observedNo commands run in the source
- DependenciesPinnedNothing floats
- SecretsNone foundNo keys in source
Five audit layers
| Layer | What we check | Result | Findings |
|---|---|---|---|
L0 Provenance | Repo owner, licence, archive status, activity | PASS | 0 |
L1 Static source | Secrets, path traversal, supply chain, hidden code | NA | 0 |
L2 Instructions | Prompt injection, authority framing, hidden directives | PASS | 0 |
L3 Inventory | The files and declarations match the source | PASS | 0 |
L4 Sandbox | Run in isolation and watch behaviour | SKIPPED why | 0 |
Gates applied: no behavioural pass.
Findings (0)
No findings at any severity. We re-audit when the source changes.
Audit 0.4.1 · source sha 33087d8174de · 14 Nov 2026Report an issue or request a re-scan
Provenance
Who published it, and what the verdict is pinned to
- Publisherhuggingface GitHub organizationOwner of the repository this skill is read from
- Source folderhuggingface/skills/skills/trl-trainingThe exact folder we audited
- LicenceApache-2.0As the repository states it
- Commit pinned for this verdict33087d8174deA new audit runs when the source changes
- Repository statusNot archivedPushed today
- Sandboxed runNot runSkills are read, not executed; layer L4 is skipped
What it tells the agent
Full source on GitHub ↗Its description: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
---
name: trl-training
description: Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
license: Apache-2.0
metadata:
author: huggingface
documentation: https://huggingface.co/docs/trl
---
# TRL
Each method pairs a `*Trainer` class with a `*Config` dataclass. Configs extend `transformers.TrainingArguments`, so all of its arguments work in any trainer config.
| Trainer | Dataset type |
|---|---|
| `SFTTrainer` | language modeling or prompt-completion |
| `DPOTrainer` | preference (chosen/rejected pairs) |
| `GRPOTrainer` | prompt-only + reward function(s) |
| `DistillationTrainer` | prompt-only + a teacher model (on-policy distillation) |
| `KTOTrainer` | unpaired preference (per-sample bool label) |
| `RewardTrainer` | preference (chosen/rejected pairs); trains a scalar reward model, not a policy |
Many more trainers (OnlineDPO, ORPO, CPO, GKD, ...) live in `trl.experimental` with unstable APIs: https://huggingface.co/docs/trl/experimental_overview
```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B", # model ID or a PreTrainedModel instance
args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()
```
Pass `model` as a string and route loading kwargs through `model_init_kwargs` (e.g. `{"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}`) instead of calling `from_pretrained` yourself. The tokenizer/processor is inferred from the model; pass `processing_class` only when it differs. For LoRA, pass `peft_config=LoraConfig(...)`.
## Dataset formats
Conversational: `{"messages": [{"role": ..., "content": ...}]}` (language modeling) or `{"prompt": [...], "completion": [...]}`. The chat template is applied automatically — never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats
## SFT: the fields that matter
```python
SFTConfig(
max_length=1024, # truncation length; None disables truncation
packing=True, # pack sequences into max_length blocks: fewer pad tokens, higher throughput
padding_free=True, # flatten batch, no padding; requires FlashAttention; implied by packing
use_liger_kernel=True, # fused Liger kernels, reduces peak memory
assistant_only_loss=True, # loss only on assistant turns (conversational datasets)
)
```
## GRPO: online RL
```python
def reward_len(completions, **kwargs):
return [-abs(20 - len(c[0]["content"])) for c in completions]
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=reward_len, # or a list; rewards are summed
args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", max_completion_length=512),
train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
```
Reward functions are called with keyword arguments `prompts`, `completions`, `completion_ids`, `trainer_state`, plus every extra dataset column — accept `**kwargs` for the ones you ignore. Return `list[float]`, one reward per completion. With conversational data, `completions` is a list of message lists, not strings.
The generation batch is `per_device_train_batch_size × num_processes × steps_per_generation` (or set `generation_batch_size` directly) and must be divisible by `num_generations` (default 8). Generation is the usual bottleneck — enable vLLM with `use_vllm=True`: `vllm_mode="colocate"` shares the training GPUs (size with `vllm_gpu_memory_utilization`); `vllm_mode="server"` uses a separate `trl vllm-serve --model <model_id>`.
`AsyncGRPOTrainer` (`trl.experimental.async_grpo`) implements the same algorithm with generation decoupled from training: a background worker streams completions from a vLLM server while the training loop consumes them, so the two overlap instead of alternating.
## CLI
Flags mirror the config fields: `trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara`. YAML via `--config`; distributed presets via `--accelerate_config zero3` (Python scripts: `accelerate launch train.py`).Excerpt, first 4,312 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.
Install
Pinned to the audited commit 33087d8
git clone https://github.com/huggingface/skills git -C skills checkout 33087d8174de
The author documents no install command, so this uses the repository directly. Our verdict covers commit 33087d8174de only.
Works in
The author has not said which agents this skill supports, and we have not tested it in any.
Audit history
Every audit, every source change
- ✓SAFE B 89 / 100
first audit · source
33087d8174de
README badge
For maintainers · shows the latest audit
[](https://modelsatlas.com/skills/huggingface/trl-training/)
The badge shows our verdict and grade for this skill, and links back to this page. It changes when the audit does.
Similar skills
mattpocock/skills
Implement a piece of work based on a spec or set of tickets.
mattpocock/skills
Compact the current conversation into a handoff document for another agent to pick up.
mattpocock/skills
Pursue a long-running goal in a single session by co-ordinating subagents.
Add this badge to your README
[](https://modelsatlas.com/skills/huggingface/trl-training/)
<a href="https://modelsatlas.com/skills/huggingface/trl-training/"><img src="https://modelsatlas.com/skills/huggingface/trl-training/badge.svg" alt="ModelsAtlas: SAFE B"></a>
Questions about trl-training
What does the trl-training skill do?
Is trl-training safe to install?
What can trl-training access on my machine?
How current is this page?
Sources
- github.com/huggingface/skills/tree/main/skills/trl-training — the audited source at commit 33087d8174de and every finding aboveObserved · 14 Nov 2026
Spotted something wrong? Report a correction · Methodology
Keep going in your AI
Opens with this page as the source. We add nothing else to the prompt.
- What does the trl-training skill do?
- Is trl-training safe to install?
- What can trl-training access on my machine?