# Upgrades (/platform/self-hosting/upgrades)

Choose and pin a release, back up the database, and rehearse the migration before upgrading. Schema migrations are forward-only, so some rollbacks require a database restore and lose writes made after the backup.

## Pin your version \[#pin-your-version]

Warning:

Pin a release tag or digest for production. `latest` can change between pulls; restarting with a pull policy that refreshes it can introduce an unplanned version and migration.

Sim publishes images to GHCR, tagged by release alongside `latest`:

```
ghcr.io/simstudioai/simstudio
ghcr.io/simstudioai/realtime
ghcr.io/simstudioai/migrations
```

Note:

Keep `app`, `realtime`, and `migrations` on the same release. They share a schema; mismatched versions can expect missing columns or incompatible behavior. An application-only rollback needs verified compatibility with the migrated schema.

Tab: Kubernetes (Helm)

```yaml
app:
  image:
    tag: "v0.9.17"  # a tag from the releases page
realtime:
  image:
    tag: "v0.9.17"  # a tag from the releases page
migrations:
  image:
    tag: "v0.9.17"  # a tag from the releases page
```

When `image.tag` is unset it uses the selected chart's `appVersion`. Published chart `1.11.8` uses `v0.9.17`; local checkout metadata can differ. Setting tags explicitly decouples image selection from chart upgrades. For maximum determinism, pin `image.digest` instead:

```yaml
app:
  image:
    digest: "sha256:..."
```

Tab: Docker Compose

Images track `latest` by default. To pin, set `SIM_VERSION` in `.env` to a tag from the [releases page](https://github.com/simstudioai/sim/releases):

```bash
# .env
SIM_VERSION=v0.9.17
```

One variable drives all three schema-coupled images, so they cannot drift apart. The `cron` service is deliberately excluded — it only makes HTTP calls and shares no schema, so it tracks `latest` unless you pin `SIM_CRON_VERSION`.

## Upgrade procedure \[#upgrade-procedure]

Before applying an upgrade, configure a direct, non-pooled `MIGRATION_DATABASE_URL` on the migrator if the app uses a pooler. Session advisory locks do not survive transaction pooling. Pass the URL through the migrator's database Secret on Helm or its explicit environment on Compose; `app.env` or `.env` alone does not reach it. See [Migration environment](#migration-environment) for the exact setup.

### Read the release notes \[#read-the-release-notes]

Check the [releases page](https://github.com/simstudioai/sim/releases) for new required environment variables and breaking changes. When the chart's minor version moves, also read its `README.md` upgrade notes — chart upgrades occasionally rename or remove values keys.

### Take a backup \[#take-a-backup]

Snapshot the database immediately before upgrading. Because migrations are forward-only, this snapshot is your only rollback path for schema changes.

```bash
# Managed Postgres — take a manual snapshot
aws rds create-db-snapshot --db-instance-identifier sim-db \
  --db-snapshot-identifier "sim-pre-upgrade-$(date +%Y%m%d)"

# Bundled Postgres — the Helm chart's database is named `sim` by default
# (Docker Compose uses `simstudio`; the cloud example values files override to `simstudio`)
kubectl exec -n simstudio statefulset/sim-postgresql -- \
  pg_dump -U postgres -Fc sim > "pre-upgrade-$(date +%F).dump"
```

### Rehearse against real data \[#rehearse-against-real-data]

Migration surprises are usually data-shaped rather than schema-shaped, so a staging run against a copy of production data catches far more than a run against an empty database.

### Apply \[#apply]

Tab: Kubernetes (Helm)

```bash
helm upgrade sim oci://ghcr.io/simstudioai/charts/sim \
  --version 1.11.8 \
  --namespace simstudio \
  --values my-values.yaml
```

Preview changed chart values before applying. The following command requires the `helm-diff` plugin:

```bash
helm diff upgrade sim oci://ghcr.io/simstudioai/charts/sim --version 1.11.8 -n simstudio --values my-values.yaml
```

Then watch the rollout:

```bash
kubectl rollout status -n simstudio deploy/sim-app --timeout=10m
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=100
```

Tab: Docker Compose

```bash
bunx sim-setup update
# The file that started your install. The Ollama stack is not managed by
# the CLI at all — see below.
COMPOSE_FILE=docker-compose.prod.yml
# COMPOSE_FILE=docker-compose.local.yml
docker compose -f "$COMPOSE_FILE" logs migrations
```

`bunx sim-setup update` obtains the new images, then brings the stack up with
`docker compose up -d`, keeping data volumes. What it runs depends on which Compose file the
install uses:

| Install | What it runs |
| - | - |
| `docker-compose.prod.yml` | Refreshes its managed copy of the Compose file, writes `POSTGRES_PASSWORD` to `.env` if it is missing, then `docker compose pull` — the versions configured by `SIM_VERSION`, or `latest` when unset |
| `docker-compose.local.yml` | `docker compose build --pull` — rebuilds from source against refreshed base images, no pull of published images |

Inspect the result with `bunx sim-setup logs`, which targets whichever Compose file the install uses.

`docker-compose.prod.yml` requires `POSTGRES_PASSWORD`. If you manage the file yourself and Compose stops with `required variable POSTGRES_PASSWORD is missing a value`, confirm which password initialized the existing database volume. If it was never set, the old default was `postgres`; restore that value first, then see [Postgres on Compose](/platform/self-hosting/security#postgres-on-compose) to rotate it.

Before updating a pinned production install, replace `SIM_VERSION` in `.env` with the chosen release tag. The update command pulls that configured tag; it does not automatically advance a pin.

The CLI detects only those two files. An install started from `docker-compose.ollama.yml` is invisible to it: `update`, `logs`, and `status` report no install, or — in a source checkout that also carries per-application env files — report that checkout's dev install instead. Upgrade that stack directly:

```bash
# That file builds the app, realtime server, and migrator from source, so the
# upgrade is a checkout plus a rebuild — pulling only refreshes Ollama and
# Postgres. The Ollama service is profile-scoped, so pass the profile you
# installed with or Compose silently skips it.
PROFILE=gpu  # or cpu
git pull
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" build --pull
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d
```

Warning:

`sim-setup update` refuses two install kinds outright rather than doing something surprising:

- **Kubernetes** — it does not upgrade Helm releases. Use `helm upgrade` after reading the chart and release notes.
- **Source / dev** — the `dev` mode that manages only Postgres and Redis. Update the checkout with git, run `bun install`, and restart `bun run dev:full`. A source checkout running `docker-compose.local.yml` is a Compose install and does update, by rebuilding.

On a standalone `docker-compose.prod.yml` install it also refuses to overwrite its managed Compose file if you have edited it by hand, comparing the file against the hash it recorded in `.sim-setup.json`: preserve or remove your customizations first.

Compose does not provide rolling updates. Plan a maintenance window while containers restart. Kubernetes supports rolling deployments, but database compatibility and in-flight execution behavior still need a rehearsal.

### Verify \[#verify]

Run the [verification checklist](/platform/self-hosting/verify). At minimum: sign in, open a workflow, execute it, upload a file, and confirm the [background jobs](/platform/self-hosting/background-jobs) are still firing.

## Rolling back \[#rolling-back]

**Application-only rollback** is possible when no migrations ran, or when you have verified the previous app is compatible with the migrated schema. Additive migrations alone do not prove compatibility:

```bash
helm rollback sim -n simstudio
```

```bash
# Production Compose — set SIM_VERSION in .env so the pin persists, replacing
# any existing line rather than appending a second one. Substitute the previous
# release tag from the releases page.
docker compose -f docker-compose.prod.yml up -d
```

The local and Ollama stacks build the app from source and ignore `SIM_VERSION`, so roll those back by checking out the previous tag and rebuilding with the file — and, for the Ollama stack, the profile — that started the install:

```bash
git checkout "<previous-release-tag>"

# Local stack
docker compose -f docker-compose.local.yml up -d --build

# Ollama stack — the profile you installed with
PROFILE=gpu  # or cpu
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d --build
```

Never roll back below v0.9.33 once account tokens are sealed: those releases cannot read them, and every connected account would have to be reconnected. See [Account token sealing](/platform/self-hosting/security#account-token-sealing).

**Rollback after an incompatible schema change** requires restoring the pre-upgrade database backup. This loses writes made since the snapshot. Stop all writers before restoring:

1. Stop traffic and all processes that can write to the database, including the app, realtime service, scheduled jobs, and any background workers.
2. Restore the pre-upgrade database snapshot.
3. Redeploy the previous image tag on the app, realtime, and migrations services, then resume the matching background workers and scheduler.
4. Verify.

Run the verification checklist before reopening traffic. Preserve the original encryption secrets with the restored database.

## When a migration fails \[#when-a-migration-fails]

The app pods will not become ready — this is by design.

```bash
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=200
```

```bash
# Or the file that started your install.
docker compose -f docker-compose.prod.yml logs migrations
```

Common causes:

| Symptom | Cause |
| - | - |
| `permission denied to create extension "vector"` | The database user lacks superuser rights. Create the pgvector extension manually as an admin, then re-run. |
| Connection refused / timeout | `DATABASE_URL` wrong, or the database is not reachable from the pod. Check network policy and credentials. |
| `Timed out ... waiting for the migration advisory lock` | Another replica's migrator is wedged holding the lock. This replica waited 30 minutes and gave up. Find the runner that is actually stuck and read its logs. |
| SQLSTATE `55P03` — lock timeout on a large table | A long-running query is holding a lock the DDL needs. The migrator sets `lock_timeout = 5s` on DDL rather than queueing every other query on that table behind it. Drain traffic and retry during a quiet window. |
| Constraint violation | Pre-existing data conflicts with a new constraint. Capture the error, restore the pre-upgrade backup, and open an issue with the exact message. |

Do not manually edit the migrations table to skip a failed migration — the schema and Sim's expectations will diverge in ways that surface much later.

## Migration reference \[#migration-reference]

Migrations are Drizzle SQL files plus registered script (data) migrations, applied by a dedicated image in one locked session. A failure in either stage fails the migration container.

- **Kubernetes** — an **init container on the app Deployment**. Every app pod waits for migrations to complete before it starts, so a failed migration blocks the rollout instead of starting an app against a mismatched schema.
- **Docker Compose** — a one-shot `migrations` service with `restart: no` that runs before the app.

Migrations are **forward-only**. Take the backup described in the upgrade procedure before applying them; there are no down-migrations.

Warning:

Upgrading from a release older than v0.9.33 with a rolling deploy leaves old pods running next to new ones for a while. Old pods cannot read sealed account tokens, so integrations can fail on them until the rollout finishes. The `migrate` backfill waits an hour after such an upgrade before it seals stored tokens, but tokens the new pods write are sealed straight away. To avoid that window, upgrade to v0.9.33 first, stop the old pods before starting the new ones, or set `ENCRYPT_ACCOUNT_TOKENS=false` for the upgrade and remove it once every pod runs the new release.

Migrations apply in sequence, so skipping several releases at once generally works mechanically. It is riskier — those combinations get less testing, and you take several releases' worth of behavior changes in one step. Upgrade in smaller increments where you can.

### How concurrent replicas are handled \[#how-concurrent-replicas-are-handled]

Every app replica runs the full migrator — there is no leader. They are serialized by a **Postgres session advisory lock**, taken with a bounded try-lock loop rather than a blocking wait:

1. One runner takes the lock and applies the pending files. The rest retry every 5 seconds.
2. The applied files are journaled, so the runners that follow find nothing to do and exit.
3. A runner that cannot get the lock within **30 minutes** fails with a message naming the deadline.

A stuck runner holds up later replicas until their 30-minute lock-acquisition deadline. Inspect the earliest migration container's logs to find the holder; a later timeout may come from a runner that was only waiting.

### Migration environment \[#migration-environment]

These control the migrator and are set on the `migrations` container, not the app.

| Variable | Purpose |
| - | - |
| `MIGRATION_DATABASE_URL` | A **direct, non-pooled** database connection URL (DSN). Falls back to `DATABASE_URL`. Setting it also arms the backend-pid check that aborts the run if the session — and with it the advisory lock — is silently recycled mid-migration; with only `DATABASE_URL` set, that check is skipped |
| `ENCRYPTION_KEY`, `ENCRYPT_ACCOUNT_TOKENS` | Let the migrator seal account tokens stored before [account token sealing](/platform/self-hosting/security#account-token-sealing) was on. The Helm init container reads both from the app Secret, and the Compose `migrations` services pass the `.env` values. Without a usable key the step stays pending and does nothing |

Warning:

Set `MIGRATION_DATABASE_URL` if anything sits between Sim and Postgres. Session advisory locks and session-level `SET`s do not survive PgBouncer transaction pooling — the lock silently does not hold, and consecutive statements can land on different backends. Point it at the database directly, bypassing the pooler.

Supply the migration URL separately from the app URL:

| Deployment | How to pass `MIGRATION_DATABASE_URL` |
| - | - |
| Helm | The init container reads the database Secret, plus only `ENCRYPTION_KEY` and `ENCRYPT_ACCOUNT_TOKENS` from the app Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL |
| Compose | Add the variable to the `migrations` service's explicit `environment:` list. Setting it in `.env` alone does not pass it to that service |
| Separate migration run | Run the migration image yourself with a direct DSN before upgrading |

Behavior that is fixed in the migrator and not configurable, but that explains what you see in the logs:

| Behavior | Value |
| - | - |
| Advisory-lock acquire deadline | 30 minutes, retried every 5s |
| `lock_timeout` on the migration session | 5 seconds for ordinary DDL — a statement that waits longer fails with SQLSTATE `55P03`. An individual migration can set it to `0` for statements that must not be interrupted, such as `CREATE INDEX CONCURRENTLY`. `statement_timeout` is `0` on the same session by default, so a long migration is not cut off once it holds its locks — though a migration can narrow it for its own statements, as `0076_damp_vector.sql` does with a 180-second local setting |
| SQL migration retry budget | 20 minutes, with exponential backoff and jitter, and only for a lock timeout (`55P03`). Script (data) migrations run once — a failure there exits immediately. They also run with `lock_timeout = 0` and `statement_timeout = 0`, so one blocked on an app-held lock waits indefinitely: a stuck script migration looks like a hung migration container, not a `55P03` |
| Connection attempts at startup | 10, for transient failures — `53300` (`too_many_connections`), `53400`, the `08xxx` connection-exception class, and socket-level errors such as `ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `EHOSTUNREACH`, and `ENOTFOUND`. A database that is simply down is retried with backoff, not failed immediately |

The `lock_timeout` deliberately trades a table-wide stall for a failed migration: without it, DDL waiting on an `AccessExclusiveLock` queues every other query on that table behind it for the whole wait.

## Related docs \[#related-docs]

- [Verify your install](/platform/self-hosting/verify)
- [Security and hardening](/platform/self-hosting/security)