Hybrid Cloud OutboxesCAUTION

Developer-first error tracking and performance monitoring

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: hybrid-cloud-outboxes
description: >-
  Guide for creating and maintaining outbox-based eventually consistent operations
  in Sentry. Most commonly used for cross-silo data replication, but applicable
  anywhere eventual consistency is needed — including single-silo deferred side
  effects, audit logging, and event fanout. Use when asked to "add outbox",
  "add outbox replication", "replicate model to control silo", "replicate model
  to cell", "add outbox category", "write outbox signal receiver", "debug stuck
  outboxes", "outbox not processing", "data not replicating", "test outbox",
  "migrate model to use outboxes", "backfill outbox data", "outbox coalescing",
  "ReplicatedCellModel", "ReplicatedControlModel", "OutboxCategory",
  "OutboxScope", or "outbox_runner". Covers model mixins, category registration,
  signal receivers, testing, backfill, and debugging workflows.
---

# Hybrid Cloud Outboxes

Sentry uses a **transactional outbox pattern** for eventually consistent operations. When a model changes, an outbox row is written inside the same database transaction. After the transaction commits, the outbox is drained — firing a signal that triggers side effects such as RPC calls, tombstone propagation, or audit logging.

The most common use case is **cross-silo data replication**: a model saved in the Cell silo produces a `CellOutbox` that, when processed, replicates data to the Control silo (or vice versa via `ControlOutbox`). But the pattern is general — outboxes work for any operation that should happen reliably after a transaction commits, even within a single silo.

There are two outbox types corresponding to the two directions of flow:

- **`CellOutbox`** — written in a Cell silo, processed in the Cell silo to push data toward Control (via RPC calls in signal receivers).
- **`ControlOutbox`** — written in the Control silo, processed in the Control silo to push data toward one or more Cell silos. Each `ControlOutbox` row targets a specific `cell_name`.

## Critical Constraints

> **Outboxes MUST be written in the same transaction as the data change.**
> The mixin classes (`ReplicatedCellModel`, `ReplicatedControlModel`) enforce this automatically via `prepare_outboxes()`. If you write outboxes manually, always use `outbox_context(transaction.atomic(...))`.

> **Handlers MUST be idempotent.**
> Outboxes can be retried on failure and are coalesced — the handler may receive only the latest version of a change, or be called multiple times for the same change.

> **`drain_shard()` MUST NOT run inside a transaction.**
> It acquires `SELECT FOR UPDATE` locks and processes messages one at a time. Calling it inside a transaction will deadlock or hold locks for too long.

> **Only the latest payload survives coalescing.**
> Multiple outbox writes for the same `(scope, shard_identifier, category, object_identifier)` are coalesced — only the row with the highest ID is processed. Never rely on every intermediate payload being delivered.

> **Every `OutboxCategory` must be registered to exactly one `OutboxScope`.**
> An assertion at import time enforces this. A category registered to zero or multiple scopes causes an import crash.

> **Bulk operations must use the producing manager.**
> Use `MyModel.objects.bulk_create()` / `bulk_update()` / `bulk_delete()` from `CellOutboxProducingManager` or `ControlOutboxProducingManager`. Raw querysets bypass outbox creation.

> **Snowflake ID models cannot use `bulk_create`.**
> The producing manager pre-allocates IDs via `SELECT nextval(...)`, which conflicts with snowflake ID generation. Use individual `save()` calls instead.

## Step 1: Determine What You Need

| Intent                                                      | Go to               |
| ----------------------------------------------------------- | ------------------- |
| Add outbox replication to a new model                       | Step 2              |
| Add a new `OutboxCategory` (not tied to a replicated model) | Step 3              |
| Write a manual signal receiver (not using model mixins)     | Step 4              |
| Migrate an existing model to use outboxes                   | Step 5, then Step 6 |
| Set up a backfill for existing data                         | Step 6              |
| Test outbox-based replication                               | Step 7              |
| Debug stuck or unprocessed outboxes                         | Step 8              |

## Step 2: Add Outbox Replication to a New Model

### 2.1 Choose the Mixin

| Data lives in... | Replicates toward... | Mixin                    | Outbox type     |
| ---------------- | -------------------- | ------------------------ | --------------- |
| Cell silo        | Control silo         | `ReplicatedCellModel`    | `CellOutbox`    |
| Control silo     | Cell silo(s)         | `ReplicatedControlModel` | `ControlOutbox` |

### 2.2 `ReplicatedCellModel` Template

Use this when a Cell model needs to replicate data to the Control silo.

```python
from sentry.backup.scopes import RelocationScope
from sentry.db.models import (
    FlexibleForeignKey,
    Model,
    cell_silo_model,
    sane_repr,
)
from sentry.db.models.manager.base_query_set import BaseQuerySet
from sentry.hybridcloud.outbox.base import ReplicatedCellModel, CellOutboxProducingManager
from sentry.hybridcloud.outbox.category import OutboxCategory


class MyModelManager(CellOutboxProducingManager["MyModel"]):
    """Manager that ensures bulk operations create outboxes."""
    pass


@cell_silo_model
class MyModel(ReplicatedCellModel):
    __relocation_scope__ = RelocationScope.Organization

    # Required: the OutboxCategory for this model (must already be registered)
    category = OutboxCategory.MY_MODEL_UPDATE

    # Use the producing manager for bulk operation support
    objects: ClassVar[MyModelManager] = MyModelManager()

    # Model fields...
    organization = FlexibleForeignKey("sentry.Organization")
    name = models.CharField(max_length=128)

  

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (2)

.claude/skills

api-docs/.node-version

Gates applied: no_behavioural_pass.