Upgrades

Choose and pin a release, back up the database, and rehearse the migration before upgrading. Schema migrations are forward-only, so some rollbacks require a database restore and lose writes made after the backup.

Pin your version

Sim publishes images to GHCR, tagged by release alongside latest:

text

ghcr.io/simstudioai/simstudio
ghcr.io/simstudioai/realtime
ghcr.io/simstudioai/migrations

@layer base { .docs-root [data-docs-tab-panel][hidden] { display: block !important; } } .docs-root [data-docs-tabs-root] [role="tablist"], .docs-root [data-docs-tabs-select] { display: none; }

Code

app:
  image:
    tag: "v0.9.17"  # a tag from the releases page
realtime:
  image:
    tag: "v0.9.17"  # a tag from the releases page
migrations:
  image:
    tag: "v0.9.17"  # a tag from the releases page

When image.tag is unset it uses the selected chart's appVersion. Published chart 1.11.8 uses v0.9.17; local checkout metadata can differ. Setting tags explicitly decouples image selection from chart upgrades. For maximum determinism, pin image.digest instead:

Code

app:
  image:
    digest: "sha256:..."

Images track latest by default. To pin, set SIM_VERSION in .env to a tag from the releases page:

Code

# .env
SIM_VERSION=v0.9.17

One variable drives all three schema-coupled images, so they cannot drift apart. The cron service is deliberately excluded — it only makes HTTP calls and shares no schema, so it tracks latest unless you pin SIM_CRON_VERSION.

Upgrade procedure

Before applying an upgrade, configure a direct, non-pooled MIGRATION_DATABASE_URL on the migrator if the app uses a pooler. Session advisory locks do not survive transaction pooling. Pass the URL through the migrator's database Secret on Helm or its explicit environment on Compose; app.env or .env alone does not reach it. See Migration environment for the exact setup.

Read the release notes
Check the releases page for new required environment variables and breaking changes. When the chart's minor version moves, also read its README.md upgrade notes — chart upgrades occasionally rename or remove values keys.

Take a backup

Snapshot the database immediately before upgrading. Because migrations are forward-only, this snapshot is your only rollback path for schema changes.

Code

# Managed Postgres — take a manual snapshot
aws rds create-db-snapshot --db-instance-identifier sim-db \
  --db-snapshot-identifier "sim-pre-upgrade-$(date +%Y%m%d)"

# Bundled Postgres — the Helm chart's database is named `sim` by default
# (Docker Compose uses `simstudio`; the cloud example values files override to `simstudio`)
kubectl exec -n simstudio statefulset/sim-postgresql -- \
  pg_dump -U postgres -Fc sim > "pre-upgrade-$(date +%F).dump"

Rehearse against real data
Migration surprises are usually data-shaped rather than schema-shaped, so a staging run against a copy of production data catches far more than a run against an empty database.

Apply

@layer base { .docs-root [data-docs-tab-panel][hidden] { display: block !important; } } .docs-root [data-docs-tabs-root] [role="tablist"], .docs-root [data-docs-tabs-select] { display: none; }

Code

helm upgrade sim oci://ghcr.io/simstudioai/charts/sim \
  --version 1.11.8 \
  --namespace simstudio \
  --values my-values.yaml

Preview changed chart values before applying. The following command requires the helm-diff plugin:

Code

helm diff upgrade sim oci://ghcr.io/simstudioai/charts/sim --version 1.11.8 -n simstudio --values my-values.yaml

Then watch the rollout:

Code

kubectl rollout status -n simstudio deploy/sim-app --timeout=10m
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=100

Code

bunx sim-setup update
# The file that started your install. The Ollama stack is not managed by
# the CLI at all — see below.
COMPOSE_FILE=docker-compose.prod.yml
# COMPOSE_FILE=docker-compose.local.yml
docker compose -f "$COMPOSE_FILE" logs migrations

bunx sim-setup update obtains the new images, then brings the stack up with docker compose up -d, keeping data volumes. What it runs depends on which Compose file the install uses:

Install	What it runs
`docker-compose.prod.yml`	Refreshes its managed copy of the Compose file, writes `POSTGRES_PASSWORD` to `.env` if it is missing, then `docker compose pull` — the versions configured by `SIM_VERSION`, or `latest` when unset
`docker-compose.local.yml`	`docker compose build --pull` — rebuilds from source against refreshed base images, no pull of published images

Inspect the result with bunx sim-setup logs, which targets whichever Compose file the install uses.

docker-compose.prod.yml requires POSTGRES_PASSWORD. If you manage the file yourself and Compose stops with required variable POSTGRES_PASSWORD is missing a value, confirm which password initialized the existing database volume. If it was never set, the old default was postgres; restore that value first, then see Postgres on Compose to rotate it.

Before updating a pinned production install, replace SIM_VERSION in .env with the chosen release tag. The update command pulls that configured tag; it does not automatically advance a pin.

The CLI detects only those two files. An install started from docker-compose.ollama.yml is invisible to it: update, logs, and status report no install, or — in a source checkout that also carries per-application env files — report that checkout's dev install instead. Upgrade that stack directly:

Code

# That file builds the app, realtime server, and migrator from source, so the
# upgrade is a checkout plus a rebuild — pulling only refreshes Ollama and
# Postgres. The Ollama service is profile-scoped, so pass the profile you
# installed with or Compose silently skips it.
PROFILE=gpu  # or cpu
git pull
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" build --pull
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d

Compose does not provide rolling updates. Plan a maintenance window while containers restart. Kubernetes supports rolling deployments, but database compatibility and in-flight execution behavior still need a rehearsal.

Verify
Run the verification checklist. At minimum: sign in, open a workflow, execute it, upload a file, and confirm the background jobs are still firing.

Rolling back

Application-only rollback is possible when no migrations ran, or when you have verified the previous app is compatible with the migrated schema. Additive migrations alone do not prove compatibility:

Code

helm rollback sim -n simstudio

Code

# Production Compose — set SIM_VERSION in .env so the pin persists, replacing
# any existing line rather than appending a second one. Substitute the previous
# release tag from the releases page.
docker compose -f docker-compose.prod.yml up -d

The local and Ollama stacks build the app from source and ignore SIM_VERSION, so roll those back by checking out the previous tag and rebuilding with the file — and, for the Ollama stack, the profile — that started the install:

Code

git checkout "<previous-release-tag>"

# Local stack
docker compose -f docker-compose.local.yml up -d --build

# Ollama stack — the profile you installed with
PROFILE=gpu  # or cpu
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d --build

Never roll back below v0.9.33 once account tokens are sealed: those releases cannot read them, and every connected account would have to be reconnected. See Account token sealing.

Rollback after an incompatible schema change requires restoring the pre-upgrade database backup. This loses writes made since the snapshot. Stop all writers before restoring:

Stop traffic and all processes that can write to the database, including the app, realtime service, scheduled jobs, and any background workers.
Restore the pre-upgrade database snapshot.
Redeploy the previous image tag on the app, realtime, and migrations services, then resume the matching background workers and scheduler.
Verify.

Run the verification checklist before reopening traffic. Preserve the original encryption secrets with the restored database.

When a migration fails

The app pods will not become ready — this is by design.

Code

kubectl logs -n simstudio deploy/sim-app -c migrations --tail=200

Code

# Or the file that started your install.
docker compose -f docker-compose.prod.yml logs migrations

Common causes:

Symptom	Cause
`permission denied to create extension "vector"`	The database user lacks superuser rights. Create the pgvector extension manually as an admin, then re-run.
Connection refused / timeout	`DATABASE_URL` wrong, or the database is not reachable from the pod. Check network policy and credentials.
`Timed out ... waiting for the migration advisory lock`	Another replica's migrator is wedged holding the lock. This replica waited 30 minutes and gave up. Find the runner that is actually stuck and read its logs.
SQLSTATE `55P03` — lock timeout on a large table	A long-running query is holding a lock the DDL needs. The migrator sets `lock_timeout = 5s` on DDL rather than queueing every other query on that table behind it. Drain traffic and retry during a quiet window.
Constraint violation	Pre-existing data conflicts with a new constraint. Capture the error, restore the pre-upgrade backup, and open an issue with the exact message.

Do not manually edit the migrations table to skip a failed migration — the schema and Sim's expectations will diverge in ways that surface much later.

Migration reference

Migrations are Drizzle SQL files plus registered script (data) migrations, applied by a dedicated image in one locked session. A failure in either stage fails the migration container.

Kubernetes — an init container on the app Deployment. Every app pod waits for migrations to complete before it starts, so a failed migration blocks the rollout instead of starting an app against a mismatched schema.
Docker Compose — a one-shot migrations service with restart: no that runs before the app.

Migrations are forward-only. Take the backup described in the upgrade procedure before applying them; there are no down-migrations.

Migrations apply in sequence, so skipping several releases at once generally works mechanically. It is riskier — those combinations get less testing, and you take several releases' worth of behavior changes in one step. Upgrade in smaller increments where you can.

How concurrent replicas are handled

Every app replica runs the full migrator — there is no leader. They are serialized by a Postgres session advisory lock, taken with a bounded try-lock loop rather than a blocking wait:

One runner takes the lock and applies the pending files. The rest retry every 5 seconds.
The applied files are journaled, so the runners that follow find nothing to do and exit.
A runner that cannot get the lock within 30 minutes fails with a message naming the deadline.

A stuck runner holds up later replicas until their 30-minute lock-acquisition deadline. Inspect the earliest migration container's logs to find the holder; a later timeout may come from a runner that was only waiting.

Migration environment

These control the migrator and are set on the migrations container, not the app.

Variable	Purpose
`MIGRATION_DATABASE_URL`	A direct, non-pooled database connection URL (DSN). Falls back to `DATABASE_URL`. Setting it also arms the backend-pid check that aborts the run if the session — and with it the advisory lock — is silently recycled mid-migration; with only `DATABASE_URL` set, that check is skipped
`ENCRYPTION_KEY`, `ENCRYPT_ACCOUNT_TOKENS`	Let the migrator seal account tokens stored before account token sealing was on. The Helm init container reads both from the app Secret, and the Compose `migrations` services pass the `.env` values. Without a usable key the step stays pending and does nothing

Supply the migration URL separately from the app URL:

Deployment	How to pass `MIGRATION_DATABASE_URL`
Helm	The init container reads the database Secret, plus only `ENCRYPTION_KEY` and `ENCRYPT_ACCOUNT_TOKENS` from the app Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL
Compose	Add the variable to the `migrations` service's explicit `environment:` list. Setting it in `.env` alone does not pass it to that service
Separate migration run	Run the migration image yourself with a direct DSN before upgrading

Behavior that is fixed in the migrator and not configurable, but that explains what you see in the logs:

Behavior	Value
Advisory-lock acquire deadline	30 minutes, retried every 5s
`lock_timeout` on the migration session	5 seconds for ordinary DDL — a statement that waits longer fails with SQLSTATE `55P03`. An individual migration can set it to `0` for statements that must not be interrupted, such as `CREATE INDEX CONCURRENTLY`. `statement_timeout` is `0` on the same session by default, so a long migration is not cut off once it holds its locks — though a migration can narrow it for its own statements, as `0076_damp_vector.sql` does with a 180-second local setting
SQL migration retry budget	20 minutes, with exponential backoff and jitter, and only for a lock timeout (`55P03`). Script (data) migrations run once — a failure there exits immediately. They also run with `lock_timeout = 0` and `statement_timeout = 0`, so one blocked on an app-held lock waits indefinitely: a stuck script migration looks like a hung migration container, not a `55P03`
Connection attempts at startup	10, for transient failures — `53300` (`too_many_connections`), `53400`, the `08xxx` connection-exception class, and socket-level errors such as `ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `EHOSTUNREACH`, and `ENOTFOUND`. A database that is simply down is retried with backoff, not failed immediately

The lock_timeout deliberately trades a table-wide stall for a failed migration: without it, DDL waiting on an AccessExclusiveLock queues every other query on that table behind it for the whole wait.

Pin your version

Upgrade procedure

Read the release notes

Take a backup

Rehearse against real data

Apply

Verify

Rolling back

When a migration fails

Migration reference

How concurrent replicas are handled

Migration environment

Related docs