Satellite telemetry

Satellites relay telemetry for the network zone they run in: they receive logs, metrics and traces through local receivers and forward them, and they execute satellite-capable pull telemetry sources (a Prometheus scrape, a Kubernetes events poll) the core cannot reach and forward the records. All of this rides the satellite’s single authenticated WebSocket to the core, alongside health-check dispatch. This page is the developer reference for how that channel is built: the extension point domain plugins contribute to, the generic protocol envelopes, the backpressure and secret-delivery mechanics, and the dependency direction that keeps satellite-backend ignorant of every domain plugin. For the wider satellite protocol (enrollment, assignments, results), see Satellites architecture; for the operator view, see Satellites.

Dependency direction

satellite-backend owns the telemetry channel but must never import a domain plugin (logstream-backend, metricstream-backend) or the telemetry platform, per the platform’s dependency rules. It inverts the dependency with an extension point: it routes generic envelopes by a stable kind string and lets the owning plugin supply the handler for that kind. satellite-backend never learns what a kind means. The forward paths are owned by the stream plugins (logstream, metricstream, tracestream); the pull-execution path (telemetry-pull) is owned by the telemetry PLATFORM (telemetry-backend), which contributes its handler the same way a domain plugin does.

The agent obeys the same rule. The pure parsers a receiver needs (OTLP logs and metrics decode, Prometheus text parse, syslog framing, native JSON) were relocated out of any *-backend into leaf packages the agent may import:

@checkstack/logstream-common, @checkstack/metricstream-common and @checkstack/tracestream-common carry the per-domain parsers and normalization. The tracestream leaf also holds the satellite trace wire contract (SatelliteTraceBatchSchema, toWireSpan / fromWireSpan) and the span-timestamp clamp, so both the agent and the core handler share one shape.
@checkstack/otlp-wire is a new zero-node-builtin leaf holding the shared OTLP wire decode, so both domains and the agent depend on one implementation without pulling in a backend.

The result is that @checkstack/satellite (the agent) depends only on *-common packages, otlp-wire, and backend-api - never on a *-backend. The same direction that keeps the core clean keeps the agent shippable.

The capability extension point

satellite-backend defines satelliteCapabilityExtensionPoint. A domain plugin registers a SatelliteCapabilityHandler against it; the platform buffers registrations, so load order does not matter. The handler is the complete contract between a domain and the telemetry channel:

export interface SatelliteCapabilityHandler {
  /** The capability kind this handler owns (e.g. "logstream"). */
  kind: string;

  /**
   * Ingest one forwarded telemetry batch. Return the per-item outcome. Set
   * `retryable: true` for a TRANSIENT failure so the agent resends; omit or
   * `false` for a terminal rejection so the agent drops the batch and counts
   * the loss. A throw is treated as transient (retryable) by the WS handler.
   */
  handleTelemetryBatch?(ctx: {
    satelliteId: string;
    payload: unknown;
    /** Items the satellite dropped from its bounded buffer since the last
     * batch of this kind (a disconnect / slow-consumer episode), keyed by the
     * group the loss belongs to (a domain string the handler interprets - the
     * stream token for the forward paths, the source instance id for
     * `telemetry-pull`) so it is attributed to the exact stream. */
    droppedByGroup?: Record<string, number>;
  }): Promise<{ accepted: number; rejected: number; retryable?: boolean }>;

  /**
   * Build the capability config to push to a satellite (e.g. its bound pull
   * source instances). Called after `authenticated` and on every config-changed
   * notify. Return `null` to push nothing. Secrets MUST NOT ride this config.
   */
  buildCapabilityConfig?(ctx: { satelliteId: string }): Promise<unknown | null>;

  /** Handle a fire-and-forget status update from a satellite (no ack). */
  handleCapabilityStatus?(ctx: {
    satelliteId: string;
    payload: unknown;
  }): Promise<void>;

  /**
   * Resolve a just-in-time secret for a `capability_secret_request`. The
   * handler resolves from ITS OWN durable state and MUST validate that the
   * requested resource is bound to `satelliteId`. Return `{ payload }` with
   * the resolved secret or `{ error }` on a binding/resolution failure.
   * Secrets resolved here MUST NOT be persisted and MUST NOT ride the config.
   */
  resolveSecret?(ctx: {
    satelliteId: string;
    payload: unknown;
  }): Promise<{ payload?: unknown; error?: string }>;
}

Every method is optional: a forward-only domain implements handleTelemetryBatch; the telemetry platform’s pull handler also implements buildCapabilityConfig, handleCapabilityStatus, and resolveSecret. The payload is opaque to satellite-backend and validated by the handler, which is what lets satellite-common stay a leaf with no dependency on any domain schema.

The owning plugin contributes in its register() by resolving the extension point and registering its handler:

import { satelliteCapabilityExtensionPoint } from "@checkstack/satellite-backend";

env
  .getExtensionPoint(satelliteCapabilityExtensionPoint)
  .registerCapability(ingest.satelliteCapabilityHandler, pluginMetadata);

The registry also exposes notifyCapabilityConfigChanged({ kind, satelliteId? }) so the owning plugin can ask the core to rebuild and re-push a kind’s capability_config when its underlying data changes (for example after a pull source instance CRUD mutation). satellite-backend fans this out across pods via a broadcast domain event, so whichever pod holds the socket performs the push.

Protocol envelopes

The telemetry channel adds generic, additive envelopes to the satellite protocol in satellite-common/src/protocol.ts. Each carries a kind and an opaque payload; the router dispatches on kind.

Satellite to core:

Message	Purpose
`telemetry_batch`	A batch of normalized items to ingest. Carries `batchId` (monotonic per connection, for dedupe), `kind`, `payload`, and optional `droppedByGroup` (per-group in-transit drop counts, keyed by stream token / source instance id).
`capability_status`	A fire-and-forget status update for a kind (e.g. per-source-instance `lastRunAt` / `lastError`). No ack.
`capability_secret_request`	A just-in-time request for a kind’s secret (e.g. a pull source’s bearer field). Carries `requestId`, `kind`, and an opaque handler-validated `payload` naming the bound resource and field.

Core to satellite:

Message	Purpose
`telemetry_ack`	Acknowledges a `telemetry_batch` by `batchId`. Carries `accepted`, `rejected`, and `retryable`. Required for every batch.
`capability_config`	Pushes a kind’s configuration (e.g. the satellite’s bound pull source instances - each with its source-type id, interval, non-secret config, secret field names, and bound signals). Sent after `authenticated` and on every notify. Opaque `payload`, never secrets.
`capability_secret_response`	Replies to a `capability_secret_request` by `requestId`, with a resolved `payload` or an `error`.

The satellite advertises its capabilities on the authenticate message (and re-advertises on heartbeat so a config change converges without a reconnect) as a capabilities: string[]. Both fields are optional for version-skew safety: an older agent omits them and the core treats it as no advertised capabilities.

Local receiver routes

Each forwarding domain advertises its own receiver capability and serves its routes on the one shared receiver HTTP server (CHECKSTACK_SATELLITE_RECEIVER_PORT, default 4318). A shipper posts to the agent exactly as it would to the core push endpoints, presenting the same per-stream source token; the agent tags the batch with the token and forwards it under the telemetry kind.

Signal	Capability flag	Routes	Token	Kind
Logs	`CHECKSTACK_SATELLITE_LOG_RECEIVERS`	`/v1/logs` (OTLP), `/ingest` (native)	`ckls_`	`logstream`
Metrics	`CHECKSTACK_SATELLITE_LOG_RECEIVERS`	`/v1/metrics` (OTLP), `/ingest/metrics` (native)	`ckms_`	`metricstream`
Traces	`CHECKSTACK_SATELLITE_TRACE_RECEIVERS`	`/v1/traces` (OTLP), `/ingest/traces` (native)	`cktr_`	`tracestream`

Trace receivers are a per-signal opt-in with their own flag rather than piggybacking on the log receiver, mirroring the per-signal token and endpoint decision. Both OTLP trace decoders (protobuf and JSON) live in the common leaf, so the trace receiver answers OTLP JSON directly - there is no 415 fallback like the metric receiver’s (whose OTLP-JSON decoder still lives in its backend). The agent never verifies a source token (it has no token database); it requires only a correctly shaped token and answers success once the batch is buffered, and the core verifies the token and re-clamps the spans (see below).

Pull execution

Beyond forwarding pushed telemetry, a satellite can EXECUTE satellite-capable pull telemetry sources at the edge. This is the generic replacement for the old private metric-scrape capability: it runs any pull source type that ships a statically-linked executor, not just Prometheus scraping.

Capability flag	Capability	Kind	Owner
`CHECKSTACK_SATELLITE_TELEMETRY_PULL`	`telemetry-pull`	`telemetry-pull`	`telemetry-backend` (platform)

The mechanics on both sides:

Config push. telemetry-backend’s handler buildCapabilityConfig sends the satellite every enabled source instance bound to it, filtered to pull types that declare supportsSatellite. Each entry carries the source-type id, name, clamped interval, timeout, the non-secret config (secrets stripped via stripSecretsForRead), the names of the secret fields, and the bound signals - never a secret value.
Agent scheduler. core/satellite/src/telemetry/pull/scheduler.ts runs one timer per pushed instance on its interval, resolves the executor by qualified source-type id from telemetryPullExecutorRegistry (core/satellite/src/telemetry/pull/executor-registry.ts), runs it under an AbortController timeout, filters emitted records to the bound signals, and enqueues them as a telemetry-pull batch. A pushed instance whose source-type id has NO registered executor is a per-instance status error ("source type not available on this satellite build"), never a crash.
Re-ingestion. The handler’s handleTelemetryBatch resolves the bound source rows in one query, checks each is enabled and satelliteId-bound, and routes the records through the SAME createBoundSink the core pull path uses, so clamping, caps and folds apply identically.
Status mirroring. handleCapabilityStatus writes each instance’s lastRunAt, lastError, and consecutiveFailures back to its telemetry_sources row.

Credit window and backpressure

Forwarding is paced so a burst inside a zone cannot overrun the core or the satellite’s memory:

The agent buffers telemetry per kind in bounded, drop-oldest in-memory buffers, bucketed by the loss-attribution group (the stream token, or the source instance id). When a buffer is full, the oldest items are dropped and counted against the group they belonged to.
A credit window limits in-flight batches. The agent holds each telemetry_batch inflight until the core replies with its telemetry_ack, so it cannot outrun a slow consumer.
The ack’s retryable flag decides the agent’s next move. true (a transient failure: over-budget, sink hiccup, no handler registered yet) means keep the batch and resend under the same batchId. false (a terminal rejection: auth-rejected) means drop the batch. The rejected count is a core-side outcome the core attributes per stream itself - the agent does NOT re-count it as an in-transit drop, which would double-count it and, for a bad token, misattribute the loss to unrelated streams. A throw inside handleTelemetryBatch is treated as transient.

The count of items a satellite dropped from its buffer during a disconnect or slow-consumer episode rides the next batch of that kind as droppedByGroup, keyed by the group (stream token / source instance id) each dropped item belonged to. The handler resolves each group key to its stream and charges that stream alone, so the loss lands on the exact stream that lost data rather than being spread across every stream in the batch. The log-stream and metric-stream overview pages render the total as Dropped in transit, distinct from any core-side drop. A drop whose group key no longer resolves to a stream (an unknown or revoked token, an unbound target) is left unattributed rather than charged to another stream.

Just-in-time secret channel

An authenticated pull source (a bearer-protected exporter, a Kubernetes API token) needs a secret, but a secret must never be persisted on a satellite nor pushed in a config that re-crosses the wire on every reconnect. The capability secret channel is the generic analogue of the health-check run-secret path:

buildCapabilityConfig pushes only the source’s non-secret config plus the NAMES of its secret fields - never a secret value.
Just before a run, the agent sends a capability_secret_request naming the bound resource and field ({ instanceId, field }).
The core routes it to the handler’s resolveSecret, which validates that the instance is enabled and bound to that satelliteId, confirms the named field is actually a stored secret, resolves it from durable state, and replies with capability_secret_response carrying the value or an error.
The agent fetches secrets per field just-in-time, caches them only for the current config generation (the cache flushes on every config push), holds each value in memory only for that run, and never writes it to disk. A secret fetch failure fails the run.

The satellite names a resource it is bound to and a field of it; it never chooses an arbitrary secret. The binding is the authorization boundary.

Binding-based authorization

The two forwarding shapes are authorized by different proofs, and neither trusts the satellite to mint authority of its own:

Receiver forwarding is authorized by the stream token. The shipper hands the satellite the same per-stream source token it would send to the HTTP push endpoint (ckls_ for logs, ckms_ for metrics, cktr_ for traces). The satellite forwards it unchanged, and the domain handler verifies it exactly as the direct HTTP push does, honoring revocation. The satellite is a relay on the same authorization path, not a new trust boundary. Because the satellite’s clock is untrusted, the core re-clamps every forwarded event against its OWN receive time before storage (log timestamps via the logstream clamp, span start/end via the trace ingest pipeline’s clampSpanTimes), so a skewed agent clock can never move a stream’s timeline.
Pull execution is authorized by the source instance binding. The handler’s handleTelemetryBatch, handleCapabilityStatus, and resolveSecret accept a record, a status, or a secret only for a source instance whose bound satelliteId matches the sending satellite, so a satellite cannot forward records for an instance it was never bound to. Binding a source to a satellite in the UI is additionally gated at authoring time by assertSatellitePullBindable (core/telemetry-backend/src/satellite/binding-auth.ts), which re-checks the binding under the CALLER’s identity and throws unless they have read access to that satellite and the satellite advertises telemetry-pull - closing the SSRF-pivot where a source manager binds to a satellite they cannot see. The runtime binding check alone does not cover WHO may author the binding.

Agent pull egress guard

A pull executor that fetches over the network (Prometheus scrape, Kubernetes API) applies the same SSRF guard the core uses before it fetches: it resolves and validates the host with resolveAndValidateHost against DEFAULT_EGRESS_DENY_CIDRS, enforces a scheme guard, caps the response size, and applies a timeout. Cloud-metadata and link-local addresses are refused; reaching internal exporters on the zone’s private network is allowed by design, which is the whole point of pulling from inside the zone. Moving the pull to the satellite does not relax the guard.