Data drains

Data drains periodically export one source of retained Sim data to a destination you control. Organization owners and admins manage configuration and run history because those records expose destination identifiers and URLs. Billing-enabled deployments require Enterprise; billing-disabled deployments use the feature and scheduler configuration under Self-hosted setup.

A new drain starts with all retained history. Delivery is at least once: retries, partial acceptance, or a crash after delivery can duplicate rows. Deduplicate downstream by each source row's id.

Create and verify a drain

1. Prepare the destination and credentials. Sim does not create buckets, containers, datasets, or tables.
2. Open Settings → Organization → Data drains and click Create drain.
3. On New drain, enter a name unique within the organization, choose one source, a destination, and Every hour or Every day.
4. Fill the destination fields below and click Create.
5. Open the drain's detail page. Use Test connection, then Run now and check Recent runs and the destination's received rows.

The detail page shows the last ten runs with status, trigger, time, row count, bytes, and failure details. Destination locators are also available through API run reads. Run now requires an enabled drain; an overlapping run can be refused. The Enabled switch pauses future runs without deleting configuration or the cursor.

Sources and row format

Create separate drains for different sources. Source rows are JSON-safe NDJSON: one JSON object per line, with an id for deduplication. Destinations such as Sentinel convert these rows to their required ingestion format.

Workflow, job, and Chat-run cursors use terminal timestamps; audit and Chat cursors use creation time. Reads use the primary database and leave a five-minute stability window. This reduces late-commit gaps but does not guarantee change-data capture.

Chats are point-in-time snapshots. Later changes to messages, title, or lastSeenAt are not emitted again. An append-only archive therefore does not track every subsequent change to a conversation.

The export uses each source's row schema. Inspect representative rows before defining warehouse columns; BigQuery receives top-level fields directly. Drains do not apply a new blanket PII scrub. Workflow log masking depends on the stored data-retention redaction policy used when the data was persisted; configuring PII_URL alone does not sanitize exported history.

Destination reference

Amazon S3 and compatible stores

Writes one NDJSON object per chunk.

Objects use AES256 server-side encryption and this layout:

{prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson

The final SDK hostname is validated and its resolved address pinned. Signed request paths are preserved without redirects. An outbound gateway that cannot preserve signed targets fails closed.

Google Cloud Storage

Supply an existing Bucket, optional Prefix, and the full Service account JSON key. The service account needs storage.objects.create; storage.objects.delete lets the connection test remove its probe. Sim uses OAuth2 service-account JWT authentication and the GCS JSON API.

Object names use the S3 layout above. sim-* object metadata uses x-goog-meta-* headers.

Azure Blob Storage

Supply Account name (3–24 lowercase characters), an existing Container, optional Prefix, and an Account key with write access. Each chunk becomes an NDJSON block blob using the same object layout. Metadata keys are lowercased for Azure.

Endpoint suffix supports blob.core.usgovcloudapi.net for US Gov and blob.core.chinacloudapi.cn for China. Sim also accepts the legacy Germany suffix blob.core.cloudapi.de; verify that the particular account still uses that endpoint before selecting it.

Microsoft Sentinel

Sends JSON arrays through the Azure Monitor Logs Ingestion API into a custom Log Analytics table in your Sentinel workspace. All five drain sources use the same envelope. This destination supports Azure public cloud with Microsoft Entra client-secret authentication; it does not use the legacy workspace shared-key Data Collector API.

Prepare Azure

1. Create a custom table, for example SimLogs_CL, in the Log Analytics workspace used by Microsoft Sentinel. Use the Analytics table plan for Sentinel analytics rules.
2. Create a data collection rule (DCR) with an input stream such as Custom-SimLogs. Declare the columns below in both the input stream and the destination table. Route the stream to that workspace/table, with transformKql: "source" and outputStream: "Custom-SimLogs_CL".
3. Copy the DCR's immutable ID (dcr-…) and logs ingestion endpoint. A DCR with direct ingestion can provide its own endpoint. For Azure Private Link, use the linked data collection endpoint (DCE); self-hosted Sim must have network access and an outbound policy permitting the resolved private address.
4. Register a Microsoft Entra application, create a client secret, and assign its service principal Monitoring Metrics Publisher on the DCR. Copy the tenant ID, application client ID, and secret value. Role assignment propagation can take up to 30 minutes.

See Microsoft's Logs Ingestion API setup and DCR routing requirements. Sim does not provision these Azure resources.

Use this input stream declaration:

{
  "Custom-SimLogs": {
    "columns": [
      { "name": "TimeGenerated", "type": "datetime" },
      { "name": "Source", "type": "string" },
      { "name": "RecordId", "type": "string" },
      { "name": "DrainId", "type": "string" },
      { "name": "RunId", "type": "string" },
      { "name": "Data", "type": "dynamic" }
    ]
  }
}

Configure and verify the drain

Choose Microsoft Sentinel, then supply Logs ingestion endpoint, DCR immutable ID, Stream name, Tenant ID, Client ID, and Client secret. Enter only the endpoint origin, such as https://my-dcr.eastus-1.ingest.monitor.azure.com; Sim appends the DCR path and API version. Endpoints must use HTTPS on the Azure public-cloud ingestion domain. Sim validates DNS, pins the resolved address and refuses redirects.

Sim obtains an access token from the tenant's /oauth2/v2.0/token endpoint using client_credentials and scope https://monitor.azure.com//.default. The client secret uses the existing encrypted drain-credential storage; access tokens stay in memory for the run and are refreshed before expiry.

Test connection writes one probe event with Source: "connection_test" and Data: {"sim":"connection-test"}. Successful testing confirms Azure accepted the request, not that a row is already queryable or a Sentinel rule fired. Run the drain and verify receipt in Log Analytics; initial ingestion can take several minutes.

An audit event is delivered as:

[
  {
    "TimeGenerated": "2026-10-16T12:00:00.000Z",
    "Source": "audit_logs",
    "RecordId": "audit-event-123",
    "DrainId": "drain-123",
    "RunId": "run-123",
    "Data": {
      "id": "audit-event-123",
      "createdAt": "2026-10-15T09:00:00.000Z",
      "action": "workflow.updated",
      "actorId": "user-123"
    }
  }
]

TimeGenerated is the drain run start, allowing retained-history backfills to be ingested without changing the original event. Source timestamps remain in Data.createdAt, Data.startedAt, Data.endedAt, or Data.completedAt. Use the appropriate original timestamp for security correlation; ingestion time alone does not establish when an action happened. The destination preserves the full original row under Data and does not automatically create Sentinel analytics rules or map entities to ASIM schemas.

For audit events, query and deduplicate by source and record ID:

SimLogs_CL
| where Source == "audit_logs"
| summarize arg_max(TimeGenerated, *) by Source, RecordId
| extend EventTime = todatetime(Data.createdAt),
         ActorId = tostring(Data.actorId),
         Action = tostring(Data.action)
| order by EventTime desc

Limits and failures

Sim splits JSON arrays at 1,000,000 UTF-8 bytes and checkpoints each accepted batch before sending the next. Azure can truncate field values above 64 KB, so Sim conservatively rejects a source row whose serialized Data exceeds 64,000 UTF-8 bytes, before sending that chunk. It never truncates the row to fit. Large Chat transcripts and workflow outputs may need an object-storage drain instead. See Azure ingestion limits.

Network failures, 408, 429, and 5xx receive up to three retries after the initial attempt. Sim respects Retry-After up to 60 seconds; a longer delay fails the run for later retry rather than sending early. Invalid schema/stream (400) and missing DCR permissions (403) fail immediately. An ingestion 401 refreshes authentication once. Errors omit provider response bodies and token values.

Delivery remains at least once. Azure acceptance followed by a lost response or failed checkpoint can replay rows; successful earlier batches remain checkpointed when a later batch fails. Deduplicate using Source and RecordId, including across recreated drains. The standard hourly/daily cadence and five-minute source stability window still apply; this is not real-time streaming.

Google BigQuery

Supply Project ID, Dataset ID, Table ID, and a Service account JSON key. Domain-scoped project IDs such as example.com:my-project are accepted. The table must already exist with compatible columns for the source's top-level fields; rows are not wrapped in one JSON or STRING column. Sim sets ignoreUnknownValues: false.

The account needs roles/bigquery.dataEditor for inserts and roles/bigquery.metadataViewer for the tables.get connection probe. Sim uses OAuth2 service-account JWT authentication and tabledata.insertAll.

Each row's insertId uses {drainId}-{runId}-{sequence}-{index}; IDs exceeding 128 characters are hashed. BigQuery offers best-effort deduplication for repeated IDs for up to one minute, so duplicates remain possible during any retry. Partial insertErrors fail the chunk and report row indices; rows already accepted can be repeated. Deduplicate source IDs downstream. See BigQuery streaming behavior.

Sim's request guards are 10 MiB and 50,000 rows. The tighter source-record and chunk limits below normally apply first.

Snowflake

Supply Account, User, Warehouse, Database, Schema, Table, and an unencrypted PKCS8 RSA Private key (PEM). Prefer <orgname>-<acctname>; legacy <locator>.<region>.<cloud> identifiers are also accepted. Register the matching public key on the user with ALTER USER ... SET RSA_PUBLIC_KEY = '...'.

The user needs INSERT on the existing table and USAGE on its warehouse, database, and schema. Role is optional. Column is an existing VARIANT column, defaulting to DATA.

Sim uses key-pair JWT authentication and SQL API v2. Each chunk uses INSERT INTO "DB"."SCHEMA"."TABLE" ("col") SELECT PARSE_JSON(column1) FROM VALUES (?), ..., with one TEXT binding per row. Identifiers are quoted to preserve case. The runner polls asynchronous 202 responses. Sim guards destination JSON at 16 MiB per row; the source-record limit is the tighter 1 MiB. This is a Sim limit rather than a statement of Snowflake's current maximum.

Datadog Logs

Choose Site from us1, us3, us5, eu1, ap1, ap2, or gov. Supply a logs-write API key, not an Application key. Service defaults to sim; optional Tags are comma-separated.

Sim posts to v2 logs intake, exposes top-level row fields as log attributes, and adds sim_drain_id:, sim_run_id:, and sim_source: tags. It sets the reserved ddsource, service, ddtags, and message fields, overriding source fields with those names.

Expanded entries are split before compression. Requests are limited to 5,000,000 uncompressed bytes, 1,000 entries, and 1,000,000 bytes per entry, with a defensive 6 MiB wire-size cap. Payloads larger than 1,024 bytes use gzip.

HTTPS webhook

Supply URL, Signing secret, optional Bearer token, and optional Signature header name (default X-Sim-Signature). On Sim Cloud, the URL must use HTTPS and resolve to public addresses. Self-hosted deployments apply their outbound-network policy: approved private endpoints and loopback are available, with HTTP only where that policy permits it. Sim blocks cloud-metadata addresses and pins the resolved address for the run to prevent DNS rebinding.

Each chunk includes:

Content-Type: application/x-ndjson
User-Agent: Sim-DataDrain/1.0
X-Sim-Timestamp: <unix-seconds>
X-Sim-Signature-Version: v1
X-Sim-Signature: t=<unix-seconds>,v1=<hex-hmac-sha256>
X-Sim-Drain-Id: <drain-id>
X-Sim-Run-Id: <run-id>
X-Sim-Source: <source-wire-value>
X-Sim-Sequence: <zero-based-chunk-index>
X-Sim-Row-Count: <row-count>
Idempotency-Key: <runId>-<sequence>

Sim signs the exact string ${timestamp}.${body} with HMAC-SHA256 and the signing secret. Recompute using the original request body and reject old timestamps—for example, those older than five minutes—to limit replay. A bearer token, when configured, is sent as Authorization: Bearer <token>.

Delivery retries up to three times after the initial attempt: 500 ms, 1 s, and 2 s with ±20% jitter. Retry-After is respected on 429 and 503; non-retryable 4xx responses fail immediately.

Checkpoints, bounds, and failures

An opaque cursor advances only after acknowledged delivery. Later failures resume after the last acknowledged chunk, but a crash between remote acceptance and checkpointing can replay data. A database claim prevents overlapping runs for one drain and fences stale workers.

A queued run that reaches its work window schedules continuation. Oversized records and missing execution archives fail visibly without advancing past the incomplete record.

The webhook idempotency key covers retries of one chunk within one run. A later replay has a new run ID; partially accepted destination requests can also repeat rows. Keep source-ID deduplication even when the destination offers its own deduplication.

Pair with data retention

For a 30-day log window, Every hour or Every day provides opportunities to export before expiry; cadence alone does not guarantee every row arrives. Source deletion, workspace scope changes, late commits, and persistent failures can still leave gaps.

Your destination owns exported data lifecycle and erasure after delivery. Sim does not remove external objects when source data or a drain is deleted.

Rotate credentials or delete a drain

Destination credentials are encrypted at rest using key-rotation-aware encryption and are not returned by API reads. An API update can supply a complete replacement credential payload; omitting credentials preserves the stored secret. Source and destination type cannot be changed on an existing drain.

The current settings detail page edits Enabled only. Changing credentials there requires recreating the drain, which starts retained-history backfill again and can duplicate archived data. Use a supported API credential replacement when cursor continuity matters; do not delete the existing drain merely to rotate a key.

Delete removes configuration and run history, without deleting exported bytes. Create, update, delete, manual-run, and connection-test actions emit best-effort audit events.

Self-hosted setup

Enable the feature through enterprise configuration or explicitly set:

DATA_DRAINS_ENABLED=true
NEXT_PUBLIC_DATA_DRAINS_ENABLED=true

An explicit feature value takes precedence over ENTERPRISE_ENABLED. When disabled, the page is hidden, mutating drain requests return 404, and the dispatcher does no work. Billing-enabled deployments retain the Enterprise gate. Keep server and public feature configuration aligned; bunx sim-setup doctor checks the pair. See self-hosted enterprise setup.

The dispatcher requires scheduled calls to GET /api/cron/run-data-drains. Set the same CRON_SECRET on the app and its caller; the endpoint returns 401 when it is unset. Generate a secret if needed, then install it in both configurations:

openssl rand -hex 32

Helm's cronjobs.jobs.runDataDrains and Docker Compose's cron service call it hourly when their schedulers are enabled. If neither runs, configure an external scheduler:

curl --fail-with-body \
  -H "Authorization: Bearer $CRON_SECRET" \
  https://sim.example.com/api/cron/run-data-drains

Trigger.dev is optional. The database backend executes the registered runner with at most two active drain jobs per app process; Trigger.dev uses serialized per-drain jobs. Database claims apply in both modes. Verify scheduler calls and actual destination receipt after enabling the feature.

When upgrading to add the Sentinel destination, apply the additive database migration and deploy the updated app and all drain workers before creating Sentinel drains. Older versions do not recognize the new destination type, so creating them during a mixed-version rollout is unsupported.

Related docs

• Data retention
• Audit logs
• Background jobs