Nightly SyncCAUTION

Ongoing research training transformer models at scale

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: nightly-sync
description: Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.
when_to_use: Working on the nightly sync PR; investigating a nightly sync failure; resolving merge conflicts between main and dev; 'nightly sync failed', 'main-to-dev merge', 'sync bot'.
---

# Nightly Sync: Main to Dev

This skill is read by the automated sync bot during the nightly-sync-main-to-dev
workflow. It contains all domain knowledge for merging main into dev, resolving
conflicts, iterating on CI, and shipping the PR.

---

## Phase 1: Create the Sync Branch and Merge

### Branch Setup

1. Create branch `$BRANCH` from `origin/dev`
2. Merge: `git merge origin/main --no-edit`
3. Resolve conflicts surgically. Do NOT use global `-X theirs`, and do NOT
   blanket-checkout main's version of a shared file. Main's version may be
   taken wholesale only for files in "Files to Override from Main" below, or
   when you have identified a specific main commit that intentionally removes
   the dev-only code. For all other conflicts, combine both sides so recent
   dev-only additions remain present.

### Preserving Dev-Only Additions

Do NOT blanket-override all shared files with main's version. Dev has features
not yet in main (new classes, new modules, new tests). The merge preserves both
sides' non-conflicting additions — only intervene where there is an actual
conflict.

### Squash-Merge Chain Detection

Dev often develops features as a chain of PRs (PR1 → PR2 → PR3) where each
builds on the last. When PR1 is squash-merged to main, git sees main's squashed
version and dev's original commits as unrelated changes. A conflict resolution
that blindly picks main can silently discard PR2/PR3's improvements on dev.

After the merge, check for this pattern:

1. For each conflicted file, run `git log --oneline origin/dev -- <file>` to
   see if dev has commits that came AFTER the code main is bringing in.
2. If dev has follow-up commits (bug fixes, refactors, extensions), **favor
   dev's version** for those sections.
3. If the conflict is just main bringing in a clean copy of what dev already
   has (no follow-ups), main's version is fine.

Practical check: run `git diff origin/dev -- <file>` on conflicted files. If
dev's code was removed or reverted, investigate whether dev's version is the
more evolved one.

Real examples from PR #4291:
- `emerging_optimizers.py`: Main's version was MORE complete — it squash-merged
  dev's PRs plus added more. Taking main for that section was correct.
- `distrib_optimizer.py`: Main overwrote dev's `GroupedQuantizedTensor` support.
  Had to restore `_is_distopt_quantized_param` and the expanded
  `_expand_quantized_param_shard_for_cast` loop while keeping main's NVFP4
  additions. This required a surgical merge combining sections from both.

Key insight: squash-merge chains can go in EITHER direction. Sometimes main
is ahead (it squash-merged dev's work + more), sometimes dev is ahead (it has
follow-up PRs). Always diff both ways before deciding which version to favor.

Real example from PR #4882 / PR #4318:
- `transformer_engine.py`: main had unrelated `TEFusedMLP` refactors, while dev
  had the new `TEFusedDenseMLP` class. The sync kept the config flag and test
  but dropped the class and `gpt_layer_specs.py` selection. That is a merge
  accident: restore the dev class and selection while preserving main's
  `TEFusedMLP.as_mlp_submodule` refactor.

### Files to Override from Main

These files have known semantic conflicts where dev's versions reference args
or APIs that main removed or renamed. Take main's version with
`git checkout origin/main -- <file>`:

- `megatron/training/training.py` — references dev-only args
- `megatron/training/initialize.py` — references dev-only args
- `megatron/training/utils.py` — references dev-only args
- `megatron/training/datasets/data_samplers.py` — references dev-only args
- `megatron/core/optimizer/layer_wise_optimizer.py` — constructor signature

**Caveat for ALL overrides:** After taking main's version of any file, you
MUST run the API Mismatch Detection procedure (see below) on that file.
Taking main's caller code while keeping dev's callee implementations is the
#1 source of sync bugs.

**IMPORTANT: Do NOT take main's `pyproject.toml`, `uv.lock`, or
`docker/Dockerfile.ci.dev`.** These three files are a tightly coupled
triple — the Dockerfile's `uv sync` command must match the dependency
groups in `pyproject.toml`, and `uv.lock` must be consistent with both.
Main's versions are missing dev-only dependencies (e.g.
`fast-hadamard-transform`, correct TransformerEngine revision) and the
`--group no_pypi_wheels` flag needed to install them. Keep dev's versions
of all three files.

**IMPORTANT: `.github/CODEOWNERS` must NEVER be modified by the sync
bot under any circumstances.** Dev's CODEOWNERS is intentionally
different from main's — do not take main's version, do not merge them,
do not touch the file. If the merge produces a conflict or a non-zero
diff against `origin/dev` on this path, restore dev's version verbatim:

```
git checkout origin/dev -- .github/CODEOWNERS
```

Then verify with `git diff origin/dev -- .github/CODEOWNERS` — output
must be empty. Modifying CODEOWNERS triggers spurious reviewer
requests and conflicts with the dev team's governance; rolling back a
CODEOWNERS change after the PR lands is painful.

**NEVER manually edit `uv.lock`.** It is a machine-generated lockfile. If
it needs to change, it must be regenerated with `uv lock` inside a CUDA
container (see `.claude/skills/build-and-test/SKILL.md`).

### Git Source Reconciliation (pyproject.toml)

After keeping dev's `pyproject.toml`, check whether main has added NEW git
sources to `[tool.uv.sources]` that don't exist in dev's version. Main's
merged code may import from packages only available at specific git revisions.

1. Diff the `[tool.uv.sources]` sections:
   `git show or

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (5)

.agents/skills

.claude/skills

CLAUDE.md

examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base.sh

megatron/core/models/hybrid/CLAUDE.md

Gates applied: no_behavioural_pass.