# Observability (/platform/self-hosting/observability)

Use health checks, container logs, and infrastructure metrics to monitor your deployment. Decide where telemetry is sent before enabling tracing or raising log verbosity.

## Health endpoints \[#health-endpoints]

| Service | Endpoint | Returns |
| - | - | - |
| app | `GET /api/health` | `{"status":"ok","timestamp":"..."}` |
| realtime | `GET /health` on port 3002 | `{"status":"ok","timestamp":"...","connections":0}` |

Warning:

`/api/health` is a **liveness** signal only. It returns `200` as long as the process is serving HTTP — it does not check the database, Redis, or object storage. A healthy response does not mean the app can serve traffic successfully, so do not treat it as a dependency check. Verify dependencies with the [smoke test](/platform/self-hosting/verify) instead.

## Kubernetes probes \[#kubernetes-probes]

The chart ships probes tuned for a Next.js cold start. Defaults for the app:

| Probe | Path | Budget |
| - | - | - |
| `startupProbe` | `/` | 60 × 5s = **5 minutes** to become ready |
| `livenessProbe` | `/` | 6 × 30s = 180s of failure before restart |
| `readinessProbe` | `/` | 3 × 10s = \~30s to shift traffic |

Realtime uses `/health` on port 3002 with a 150-second startup budget.

The generous startup budget matters: a cold Next.js start on a large bundle can take minutes, and a tighter liveness probe will restart the pod mid-boot in a loop. If you customize probes, keep the startup budget well above your observed cold-start time.

```yaml
app:
  startupProbe:
    httpGet:
      path: /
      port: 3000
    periodSeconds: 5
    failureThreshold: 60
```

## Logs \[#logs]

Both services log structured JSON to stdout. Collect them with whatever you already run — Fluent Bit, Vector, Datadog Agent, Loki.

Warning:

In production builds the logger defaults to `ERROR`, and the Helm chart does not set `LOG_LEVEL` for the app or realtime. Until you raise it, the only thing in the logs is errors — which is why a healthy-looking deployment can appear to log nothing at all. Set `LOG_LEVEL: "INFO"` while commissioning a deployment or debugging.

```bash
kubectl logs -n simstudio -l app.kubernetes.io/component=app --tail=200 -f
kubectl logs -n simstudio -l app.kubernetes.io/component=realtime --tail=200 -f
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=100
```

```bash
docker compose -f docker-compose.prod.yml logs -f simstudio
```

Every API request carries a request ID that appears in all log lines for that request — the fastest way to reconstruct a failing call.

Workflow execution logs are stored in PostgreSQL and shown in the app's **Logs** view. Use them for workflow behavior and container logs for infrastructure failures.

### Redacting PII from logs \[#redacting-pii-from-logs]

Deploy the PII service, configure its internal network path, and enable the stored **Logs** redaction policy under **Settings → Organization → Data retention**. Deploying the service alone does not mask logs. Follow [Security's PII setup](/platform/self-hosting/security#pii-redaction) and [Data retention](/platform/enterprise/data-retention).

## Metrics \[#metrics]

Warning:

The default app and realtime images **do not expose a `/metrics` endpoint**. The chart's `monitoring.serviceMonitor` option exists for builds that do — enabling it against the stock images produces a ServiceMonitor that scrapes nothing.

Until an application metrics endpoint ships, build alerting from the signals that do exist:

- **Kubernetes state** — pod restarts, `CrashLoopBackOff`, OOMKills, replica count vs desired, PVC utilization (kube-state-metrics). For a point-in-time read on resource pressure, `kubectl top pods -n simstudio` shows current CPU and memory usage — compare it against the limits from `kubectl describe pod`. It is the first thing to check when a pod is being OOMKilled.
- **Ingress/load balancer** — request rate, 5xx rate, p99 latency, websocket connection count.
- **PostgreSQL** — connection count vs `max_connections`, replication lag, disk usage, long-running queries.
- **Redis** — memory usage, evictions, connected clients.
- **CronJobs** — last successful completion per job.

### CloudWatch metrics \[#cloudwatch-metrics]

Sim's hosted-key metrics recorder arms itself whenever `AWS_ACCESS_KEY_ID` is present, but every call site is additionally gated on the deployment being Sim Cloud. A self-hosted deployment emits no `PutMetricData` calls, even with AWS credentials configured for S3. If you want defence in depth against that gate ever changing, deny `cloudwatch:PutMetricData` on the IAM identity behind those credentials.

## What to alert on \[#what-to-alert-on]

| Alert | Why it matters |
| - | - |
| App pod restart loop / OOMKilled | Memory is the constraining resource; OOMKills mean executions are dying mid-run |
| A CronJob has not succeeded within its expected interval plus a chosen grace period | Scheduled workflows and polling triggers may have stopped. Choose a threshold for each job based on its schedule and expected runtime |
| Ingress 5xx rate above baseline | Broad user impact |
| Postgres connections above 80% of `max_connections` | Next replica or traffic spike will start failing |
| Postgres disk above 80% | Knowledge base embeddings grow steadily |
| Redis unreachable | Live collaboration and status updates can stop; inspect app and realtime connection errors |
| Certificate expiry within 14 days | Especially with manually managed certs |
| Object storage 4xx/5xx rate | Broken uploads usually show here first |

Alert on `kube_cronjob_status_last_successful_time` with a threshold derived from each job's schedule and expected runtime. A successful HTTP call can mean work was accepted, not completed; also verify expected executions and outputs in the app.

## Telemetry \[#telemetry]

Whether telemetry runs depends on how you deploy. A Docker Compose or source deployment enables server telemetry unless you turn it off, so a deployment with an egress policy should decide about this deliberately rather than inherit the default.

Telemetry has two halves that are decided at different times.

| Half | When it is decided | Default |
| - | - | - |
| Server SDK | Runtime | **Off** on Helm (`app.envDefaults` sets `NEXT_TELEMETRY_DISABLED: "1"`); **on** for Docker Compose and source runs, which set nothing |
| Browser | Server setting at page render, then browser preferences | Disabled when `NEXT_TELEMETRY_DISABLED=1`; otherwise waits for session resolution and the signed-in user's saved preference. Hosted Sim also requires the applicable analytics permission. |

`NEXT_TELEMETRY_DISABLED=1` also stops `POST /api/telemetry`, the relay that forwards submitted browser events, from forwarding anything. Otherwise it posts each event to `TELEMETRY_ENDPOINT` as-is, or to the endpoint in `apps/sim/telemetry.config.ts`. Because these events are unauthenticated input, the relay never sends `OTEL_EXPORTER_OTLP_HEADERS` with them.

When the server SDK runs against a collector you configure, it exports three OTLP signals to the same base endpoint: traces at `/v1/traces`, metrics at `/v1/metrics`, and application **log records** at `/v1/logs`. On the anonymous default (`https://telemetry.simstudio.ai`) it exports traces only, and never sends `OTEL_EXPORTER_OTLP_HEADERS` there. Default instrumentation records usage, performance, errors, and AI/LLM traces rather than raw workflow content. Error metadata and log records can still contain sensitive values; inspect the signals you send to your collector.

Log records at or above `LOG_LEVEL` carry structured metadata, error messages, and stack traces, which can include account IDs, emails, and URLs. Raising `LOG_LEVEL` increases that stream. Configure your collector and retention policy accordingly.

`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is **off by default**. Setting it to `true`, `1`, or `yes` enables Chat message capture, including prompts, assistant text, tool arguments, and previews. Treat it as an explicit content-sharing decision before enabling it.

`NEXT_TELEMETRY_DISABLED=1` stops the server's OpenTelemetry SDK and disables browser collection on newly loaded pages. It does not affect Trigger.dev task telemetry, configured separately below. The **Settings → General → Privacy → Allow browser telemetry** toggle stops optional browser performance and error diagnostics, discards queued events, and remembers the choice. It does not stop server spans.

## Tracing \[#tracing]

Enable both the collector and application tracing on Helm:

```yaml
telemetry:
  enabled: true
app:
  envDefaults:
    NEXT_TELEMETRY_DISABLED: null
```

Warning:

`telemetry.enabled` alone deploys a collector that receives no app signals while `NEXT_TELEMETRY_DISABLED=1`. Set the inherited key to `null` to remove it, as shown above. This also enables browser collection subject to the signed-in account's preference.

Or point the app at a collector you already run:

| Variable | Purpose |
| - | - |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector endpoint (OTLP over HTTP) |
| `OTEL_EXPORTER_OTLP_HEADERS` | Auth headers, `key=value` comma-separated |
| `TELEMETRY_ENDPOINT` | Legacy endpoint variable, still honored |
| `TELEMETRY_SAMPLING_RATIO` | Sampling ratio, `0`–`1` |
| `OTEL_TRACES_SAMPLER_ARG` | Sampling ratio, used only when `TELEMETRY_SAMPLING_RATIO` is unset |
| `OTEL_DEPLOYMENT_ENVIRONMENT` | Environment label on emitted spans |

The endpoint is resolved by precedence, first match wins: `OTEL_EXPORTER_OTLP_ENDPOINT`, then `TELEMETRY_ENDPOINT`, then the endpoint in `apps/sim/telemetry.config.ts`. This precedence governs the server SDK only — the `/api/telemetry` relay reads `TELEMETRY_ENDPOINT`, then the endpoint in `telemetry.config.ts`, and never `OTEL_EXPORTER_OTLP_ENDPOINT`. Sampling resolves the same way: `TELEMETRY_SAMPLING_RATIO`, then `OTEL_TRACES_SAMPLER_ARG`, defaulting to 1.0. Editing `telemetry.config.ts` only changes the last fallback, so an environment variable set anywhere in your stack silently wins over it.

The app does not read `OTEL_SERVICE_NAME`: server spans use the literal service name `mothership`, even when Helm sets `sim-app`. Use `mothership` in collector filters.

If you enable the chart's Jaeger export, point `telemetry.jaeger.endpoint` at Jaeger's **OTLP gRPC port (4317)** — the collector exports over OTLP.

### Trigger.dev task telemetry \[#triggerdev-task-telemetry]

`GRAFANA_OTLP_ENDPOINT`, `GRAFANA_OTLP_HEADERS`, and `GRAFANA_DEPLOYMENT_ENVIRONMENT` are read only by `apps/sim/trigger.config.ts`. They route **Trigger.dev background-task** telemetry to Grafana Cloud and have no effect on the app's own OTLP export — for that, use `OTEL_EXPORTER_OTLP_ENDPOINT` above.

| Variable | Purpose |
| - | - |
| `GRAFANA_OTLP_ENDPOINT` | Grafana OTLP gateway base URL |
| `GRAFANA_OTLP_HEADERS` | e.g. `Authorization=Basic <base64(instanceId:token)>` |
| `GRAFANA_DEPLOYMENT_ENVIRONMENT` | Deployment tier label |

Warning:

All three or none. Setting one or two throws at startup with a message naming them, so a partially configured worker will not boot.

## Related docs \[#related-docs]

- [Background jobs](/platform/self-hosting/background-jobs)
- [Verify your install](/platform/self-hosting/verify)