@render-lab/tasks-datadog

⚠️ Experimental: proof of concept. This package is part of the Render Tasks POC and is published for testing only. It is not fully tested or production ready. Task names, inputs, outputs, and behavior can change or break in any release. Pin exact versions and expect breaking changes.

Durable Datadog monitor, incident, downtime, and event tasks for Render Workflows.

This pack talks to the Datadog v1/v2 REST APIs directly rather than through a vendor SDK. An SDK retries, polls, and paginates internally, which would hide each attempt from Render Workflows. Direct HTTP keeps every poll, page fetch, and retry as one observable durable run, with the baked-in retry policy as the single source of truth (vendor retries are off). Datadog is a tool teams choose, so it ships as a vendor-first brand pack (ADR-0013).

import {
  searchMonitors,
  getMonitor,
  awaitMonitorRecovery,
  createIncident,
  getIncident,
  updateIncident,
  createIncidentTodo,
  updateIncidentTodo,
  scheduleDowntime,
  cancelDowntime,
  postEvent,
  searchEvents,
} from "@render-lab/tasks-datadog";

Install

pnpm add @render-lab/tasks-datadog @renderinc/sdk

@renderinc/sdk is a peer dependency pinned to an exact version so every task registers against the one shared TaskRegistry (ADR-0001). @render-lab/triggers is an optional peer, needed only if you wire the webhook adapter into a dispatch server.

Tasks

| Task | Input | Output | Retry | | --- | --- | --- | --- | | datadog.searchMonitors | SearchMonitorsInput | MonitorDTO[] | general (3× backoff) | | datadog.getMonitor | GetMonitorInput | MonitorDTO | general (3× backoff) | | datadog.awaitMonitorRecovery | AwaitMonitorRecoveryInput | AwaitMonitorRecoveryResult | await (poll every 15s up to 1h) | | datadog.createIncident | CreateIncidentInput | IncidentDTO | none (0 retries) | | datadog.getIncident | GetIncidentInput | IncidentDTO | general (3× backoff) | | datadog.updateIncident | UpdateIncidentInput | IncidentDTO | general (3× backoff) | | datadog.createIncidentTodo | CreateIncidentTodoInput | IncidentTodoDTO | none (0 retries) | | datadog.updateIncidentTodo | UpdateIncidentTodoInput | IncidentTodoDTO | general (3× backoff) | | datadog.scheduleDowntime | ScheduleDowntimeInput | DowntimeDTO | none (0 retries) | | datadog.cancelDowntime | CancelDowntimeInput | CancelDowntimeResult | general (3× backoff) | | datadog.postEvent | PostEventInput | EventDTO | none (0 retries) | | datadog.searchEvents | SearchEventsInput | EventPage | general (3× backoff) |

Retry and idempotency

Two policies plus one await cadence keep durability honest (ADR-0005):

| Policy | Applies to | Why | | --- | --- | --- | | DATADOG_RETRY (3× backoff ~1s, 2s, 4s) | reads and replay-safe convergent writes: searchMonitors, getMonitor, getIncident, updateIncident, updateIncidentTodo, cancelDowntime, searchEvents | Reading is safe to repeat; the incident/todo patches are absolute, so a retry re-applies the same target state; a repeated cancelDowntime normalizes a 404 to a settled cancellation and converges. | | DATADOG_NO_RETRY (0 retries) | non-idempotent creates: createIncident, createIncidentTodo, scheduleDowntime, postEvent | These have no stable request idempotency key, so a retried attempt would duplicate the remote work (a second incident, a duplicate todo/downtime/event) rather than converge. Resume by reading the resource instead. | | DATADOG_AWAIT_RETRY (poll every 15s for up to 1h) | fixed-interval polling: awaitMonitorRecovery | A durable wait: SDK 0.6.0 has no native sleep, so the task polls by throwing. |

awaitMonitorRecovery is a durable wait. It reads the monitor's overall state on each attempt: "OK" returns { monitorId, state: "OK", recovered: true }; "Unknown" throws a distinct terminal error (a human should look, not a silent forever-retry); every other state (Alert, Warn, No Data) throws a progress error so the fixed 15s-cadence retry re-polls.

Datadog site

DD_SITE (default datadoghq.com) selects the region. The REST base is https://api.${DD_SITE} and event submission targets https://event-management-intake.${DD_SITE}. Set DD_SITE to your org's site (for example datadoghq.eu, us3.datadoghq.com, ddog-gov.com).

Pagination and payload bounds

Every payload and result stays under the 4 MB Render Workflows limit. searchMonitors and searchEvents bound their page size: perPage and limit must be integers from 1 through 100. searchEvents follows the Datadog cursor for one requested page only and returns nextCursor (or null) — it never auto-drains unbounded pages. Persist nextCursor between calls to walk further.

Lifecycle examples

Open an incident, track work, and mute noisy alerts during response:

const incident = await createIncident({
  title: "Checkout latency spike",
  customerImpactScope: "checkout",
  severity: "SEV-2",
});
await createIncidentTodo({ incidentId: incident.id, content: "Page infra on-call" });

// Mute the monitor while you deploy the mitigation.
const downtime = await scheduleDowntime({ scope: "service:checkout", monitorId: 7 });

// Wait for the monitor to recover (durable poll), then resolve and unmute.
await awaitMonitorRecovery({ monitorId: 7 });
await updateIncident({ incidentId: incident.id, status: "resolved" });
await cancelDowntime({ downtimeId: downtime.id });
await postEvent({ title: "Incident resolved", text: incident.title, tags: ["team:checkout"] });

Extending and testing

Every task exports both the wrapped task and its raw *Impl (ADR-0004). The impl takes an injected DatadogDeps so you can re-register it under your own name/retry or unit-test it with a fake:

import { updateIncidentImpl, type DatadogDeps } from "@render-lab/tasks-datadog";

const deps: DatadogDeps = {
  datadog: {
    ...realPort,
    updateIncident: async (input) => ({ ...fixture, status: input.status ?? "active" }),
  } as DatadogDeps["datadog"],
};
await updateIncidentImpl({ incidentId: "inc-1", status: "resolved" }, deps);

Webhook adapter (optional)

The registration-free ./webhooks subpath exposes a shared-secret adapter. Datadog webhooks do not carry a vendor-generated cryptographic signature, so this adapter authenticates a caller-configured custom header you set on the Datadog webhook and mirror here as DATADOG_WEBHOOK_SECRET:

import { datadogSharedSecretAdapter } from "@render-lab/tasks-datadog/webhooks";

export const adapter = datadogSharedSecretAdapter({
  // headerName defaults to "x-render-webhook-secret"; secret defaults to DATADOG_WEBHOOK_SECRET
  onEvent: ({ payload }) =>
    payload.eventType === "monitor_alert"
      ? { task: "ops.page", args: [{ id: payload.id }] }
      : null,
});

Security semantics (be honest about what this proves): the adapter verifies that the request carries a shared secret both sides hold, compared in constant time with a length check (which also defeats prefix confusion). It does not prove a Datadog-generated signature, does not protect the body from mutation in transit, and provides no replay protection. Require HTTPS and a high-entropy secret. There is deliberately no verifyDatadogSignature.

Environment contract

Credentials are read lazily at first use, never at import (ADR-0007). Constructing the default port never throws — only the first task call does, close to the task that needs it.

| Variable | Required | Purpose | | --- | --- | --- | | DD_API_KEY | Yes | Lazy Datadog API authentication. | | DD_APP_KEY | Yes | Lazy Datadog application authentication. | | DD_SITE | No | Datadog site suffix. Defaults to datadoghq.com. | | DATADOG_WEBHOOK_SECRET | For the optional adapter | Exact custom-header value configured in Datadog. This is not a Datadog signature secret. |

Testing

pnpm -C packages/tasks-datadog test        # Tier 1: hermetic (no secrets, no network)
pnpm -C packages/tasks-datadog test:live   # Tier 2: live, needs RUN_LIVE=1 + real DD_* keys
pnpm -C packages/tasks-datadog build
pnpm -C packages/tasks-datadog typecheck