Observability

Use health checks, container logs, and infrastructure metrics to monitor your deployment. Decide where telemetry is sent before enabling tracing or raising log verbosity.

Health endpoints

Kubernetes probes

The chart ships probes tuned for a Next.js cold start. Defaults for the app:

Realtime uses /health on port 3002 with a 150-second startup budget.

The generous startup budget matters: a cold Next.js start on a large bundle can take minutes, and a tighter liveness probe will restart the pod mid-boot in a loop. If you customize probes, keep the startup budget well above your observed cold-start time.

app:
  startupProbe:
    httpGet:
      path: /
      port: 3000
    periodSeconds: 5
    failureThreshold: 60

Logs

Both services log structured JSON to stdout. Collect them with whatever you already run — Fluent Bit, Vector, Datadog Agent, Loki.

kubectl logs -n simstudio -l app.kubernetes.io/component=app --tail=200 -f
kubectl logs -n simstudio -l app.kubernetes.io/component=realtime --tail=200 -f
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=100

docker compose -f docker-compose.prod.yml logs -f simstudio

Every API request carries a request ID that appears in all log lines for that request — the fastest way to reconstruct a failing call.

Workflow execution logs are stored in PostgreSQL and shown in the app's Logs view. Use them for workflow behavior and container logs for infrastructure failures.

Redacting PII from logs

Deploy the PII service, configure its internal network path, and enable the stored Logs redaction policy under Settings → Organization → Data retention. Deploying the service alone does not mask logs. Follow Security's PII setup and Data retention.

Metrics

Until an application metrics endpoint ships, build alerting from the signals that do exist:

• Kubernetes state — pod restarts, CrashLoopBackOff, OOMKills, replica count vs desired, PVC utilization (kube-state-metrics). For a point-in-time read on resource pressure, kubectl top pods -n simstudio shows current CPU and memory usage — compare it against the limits from kubectl describe pod. It is the first thing to check when a pod is being OOMKilled.
• Ingress/load balancer — request rate, 5xx rate, p99 latency, websocket connection count.
• PostgreSQL — connection count vs max_connections, replication lag, disk usage, long-running queries.
• Redis — memory usage, evictions, connected clients.
• CronJobs — last successful completion per job.

CloudWatch metrics

Sim's hosted-key metrics recorder arms itself whenever AWS_ACCESS_KEY_ID is present, but every call site is additionally gated on the deployment being Sim Cloud. A self-hosted deployment emits no PutMetricData calls, even with AWS credentials configured for S3. If you want defence in depth against that gate ever changing, deny cloudwatch:PutMetricData on the IAM identity behind those credentials.

What to alert on

Alert on kube_cronjob_status_last_successful_time with a threshold derived from each job's schedule and expected runtime. A successful HTTP call can mean work was accepted, not completed; also verify expected executions and outputs in the app.

Telemetry

Whether telemetry runs depends on how you deploy. A Docker Compose or source deployment enables server telemetry unless you turn it off, so a deployment with an egress policy should decide about this deliberately rather than inherit the default.

Telemetry has two halves that are decided at different times.

NEXT_TELEMETRY_DISABLED=1 also stops POST /api/telemetry, the relay that forwards submitted browser events, from forwarding anything. Otherwise it posts each event to TELEMETRY_ENDPOINT as-is, or to the endpoint in apps/sim/telemetry.config.ts. Because these events are unauthenticated input, the relay never sends OTEL_EXPORTER_OTLP_HEADERS with them.

When the server SDK runs against a collector you configure, it exports three OTLP signals to the same base endpoint: traces at /v1/traces, metrics at /v1/metrics, and application log records at /v1/logs. On the anonymous default (https://telemetry.simstudio.ai) it exports traces only, and never sends OTEL_EXPORTER_OTLP_HEADERS there. Default instrumentation records usage, performance, errors, and AI/LLM traces rather than raw workflow content. Error metadata and log records can still contain sensitive values; inspect the signals you send to your collector.

Log records at or above LOG_LEVEL carry structured metadata, error messages, and stack traces, which can include account IDs, emails, and URLs. Raising LOG_LEVEL increases that stream. Configure your collector and retention policy accordingly.

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is off by default. Setting it to true, 1, or yes enables Chat message capture, including prompts, assistant text, tool arguments, and previews. Treat it as an explicit content-sharing decision before enabling it.

NEXT_TELEMETRY_DISABLED=1 stops the server's OpenTelemetry SDK and disables browser collection on newly loaded pages. It does not affect Trigger.dev task telemetry, configured separately below. The Settings → General → Privacy → Allow browser telemetry toggle stops optional browser performance and error diagnostics, discards queued events, and remembers the choice. It does not stop server spans.

Tracing

Enable both the collector and application tracing on Helm:

telemetry:
  enabled: true
app:
  envDefaults:
    NEXT_TELEMETRY_DISABLED: null

Or point the app at a collector you already run:

The endpoint is resolved by precedence, first match wins: OTEL_EXPORTER_OTLP_ENDPOINT, then TELEMETRY_ENDPOINT, then the endpoint in apps/sim/telemetry.config.ts. This precedence governs the server SDK only — the /api/telemetry relay reads TELEMETRY_ENDPOINT, then the endpoint in telemetry.config.ts, and never OTEL_EXPORTER_OTLP_ENDPOINT. Sampling resolves the same way: TELEMETRY_SAMPLING_RATIO, then OTEL_TRACES_SAMPLER_ARG, defaulting to 1.0. Editing telemetry.config.ts only changes the last fallback, so an environment variable set anywhere in your stack silently wins over it.

The app does not read OTEL_SERVICE_NAME: server spans use the literal service name mothership, even when Helm sets sim-app. Use mothership in collector filters.

If you enable the chart's Jaeger export, point telemetry.jaeger.endpoint at Jaeger's OTLP gRPC port (4317) — the collector exports over OTLP.

Trigger.dev task telemetry

GRAFANA_OTLP_ENDPOINT, GRAFANA_OTLP_HEADERS, and GRAFANA_DEPLOYMENT_ENVIRONMENT are read only by apps/sim/trigger.config.ts. They route Trigger.dev background-task telemetry to Grafana Cloud and have no effect on the app's own OTLP export — for that, use OTEL_EXPORTER_OTLP_ENDPOINT above.

Related docs

• Background jobs
• Verify your install