Openvla OftSAFE

Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. Maintained by Orchestra Research.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: fine-tuning-openvla-oft
description: Fine-tunes and evaluates OpenVLA-OFT and OpenVLA-OFT+ policies for robot action generation with continuous action heads, LoRA adaptation, and FiLM conditioning on LIBERO simulation and ALOHA real-world setups. Use when reproducing OpenVLA-OFT paper results, training custom VLA action heads (L1 or diffusion), deploying server-client inference for ALOHA, or debugging normalization, LoRA merge, and cross-GPU issues.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [OpenVLA, OpenVLA-OFT, VLA, Robotics, Fine-Tuning, LIBERO, ALOHA, LoRA, FiLM, Action Chunking, Deployment, Continuous Actions]
dependencies: [torch==2.2.0, transformers>=4.40.0, peft==0.11.1, draccus==0.8.0, accelerate>=0.25.0, wandb>=0.16.0, fastapi>=0.100.0, uvicorn>=0.24.0, tensorflow==2.15.0, robosuite==1.4.0]  # Exact pins: OpenVLA-OFT paper results were validated on torch==2.2.0, peft==0.11.1, tensorflow==2.15.0; upgrading torch may require re-tuning the LoRA adapter merge step and re-validating action head outputs
---

# OpenVLA-OFT

Fine-tuning and evaluation workflows for OpenVLA-OFT and OpenVLA-OFT+ from the official `openvla-oft` codebase. Covers blank-machine setup plus LoRA-based adaptation of OpenVLA for robot action generation with continuous action prediction heads.

## Quick start

Clone the public repo, follow the official setup, then evaluate a pretrained LIBERO checkpoint:

```bash
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft
python experiments/robot/libero/run_libero_eval.py \
  --pretrained_checkpoint moojink/openvla-7b-oft-finetuned-libero-spatial \
  --task_suite_name libero_spatial \
  --center_crop True \
  --num_trials_per_task 50 \
  --seed 7
```

## Core concepts

**What OpenVLA-OFT changes**: Standard OpenVLA tokenizes continuous actions into discrete bins, losing precision. OFT replaces this with dedicated continuous action heads (L1 regression or diffusion) while keeping the VLA backbone frozen and adapting via LoRA.

**OFT vs OFT+ variants**:

| Variant | FiLM | Images | Typical use |
|---------|------|--------|-------------|
| OFT | Off | 2 (front + wrist) | LIBERO simulation |
| OFT+ | On | 3 (high + left + right wrist) | ALOHA real-world |

**Key architecture choices**:
- **LoRA adaptation**: Rank-32 LoRA on VLA backbone (no full fine-tuning needed)
- **Continuous actions**: L1 regression head (default) or diffusion head
- **FiLM conditioning**: Feature-wise Linear Modulation for stronger language grounding in OFT+
- **Multi-image input**: Configurable 2 or 3 camera streams via `num_images_in_input`

## Compute requirements

| Task | GPU | VRAM | Notes |
|------|-----|------|-------|
| LIBERO evaluation | 1x A100/A40 | ~16 GB | Single GPU |
| ALOHA evaluation | 1x A100/A40 | ~18 GB | Single GPU |
| LIBERO fine-tuning | 8x A100 | ~27 GB/GPU | Paper default |
| ALOHA fine-tuning (OFT+) | 8x A100 | ~35 GB/GPU | FiLM + 3 images |
| LoRA merge | 1x any GPU | ~16 GB | One-time step |

## Expected performance benchmarks

Official results (paper setup, seed=7, 50 trials per task):

| Task Suite | Task-Specific | Combined Policy | Notes |
|-----------|--------------|-----------------|-------|
| LIBERO-Spatial | 97.2% | 96.8% | Easiest suite |
| LIBERO-Object | 97.4% | 97.0% | Object manipulation |
| LIBERO-Goal | 95.8% | 95.4% | May peak at 50k-100k steps |
| LIBERO-10 | 98.0% | 98.0% | Long-horizon tasks |
| **Average** | **97.1%** | **96.8%** | Near-equivalent |

Reproduction notes: results are tied to Python 3.10.14, PyTorch 2.2.0, NVIDIA A100, and custom Transformers fork.

## When to use vs alternatives

**Use OpenVLA-OFT when:**
- The target task is robot action generation with visual and language conditioning
- LoRA-based adaptation of `openvla/openvla-7b` is preferred
- You need official LIBERO or ALOHA workflows from the OpenVLA-OFT paper
- You want continuous action heads (L1 regression or diffusion) instead of tokenized actions

**Use alternatives when:**
- You need a different VLA architecture (use `fine-tuning-serving-openpi` for pi0/pi0.5 models)
- You need the NVIDIA Cosmos Policy stack (use `evaluating-cosmos-policy`)
- You need general LLM fine-tuning without robot action heads

---

## Workflow 1: Set up environment

Copy this checklist and track progress:

```text
Setup Progress:
- [ ] Step 1: Create conda env and install PyTorch
- [ ] Step 2: Install openvla-oft package in editable mode
- [ ] Step 3: Install FlashAttention2
- [ ] Step 4: Verify critical versions
```

**Step 1: Create conda env and clone repo**

```bash
conda create -n openvla-oft python=3.10 -y
conda activate openvla-oft
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft
pip3 install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0
pip3 install robosuite==1.4.0
```

**Step 2: Install package**

```bash
pip install -e .
```

**Step 3: Install FlashAttention2**

```bash
pip install packaging ninja
pip install "flash-attn==2.5.5" --no-build-isolation
```

**Step 4: Verify versions**

```python
import torch, transformers, peft
print(f"PyTorch: {torch.__version__}")         # Expected: 2.2.0
print(f"Transformers: {transformers.__version__}")
print(f"PEFT: {peft.__version__}")             # Expected: 0.11.1
```

---

## Workflow 2: Evaluate pretrained checkpoints on LIBERO

```text
LIBERO Eval Progress:
- [ ] Step 1: Install LIBERO dependencies
- [ ] Step 2: Choose checkpoint and task suite
- [ ] Step 3: Run evaluation
- [ ] Step 4: Parse and validate results
```

**Step 1: Install LIBERO**

```bash
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
pip install -e LIBERO
pip install -r experiments/robot/libero/libero_requirements.txt
```

**Step 2: Choose checkpoint**

| Checkpoint | Task suite |
|-----------|------------|
| `moojink/openvla-7b-oft-finetuned-libero-spatial` | `libero_spatial` |
| `moojink/openvla-7b-oft-finetuned-libero-object` | `libero_object` |
| `moojink/openvla-7b-oft-finetuned-libero-goal`

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.