Satellites

A Satellite is a lightweight Checkstack agent that runs in a different network or region from your main Checkstack instance and executes health checks on its behalf. The probe runs near the target, the result flows back over a persistent WebSocket connection, and the core records it just like a locally-executed run. This page explains when to use satellites and how the pairing works.

Note

For the protocol, enrollment handshake, and result-authorization invariant, see the developer reference: Satellites architecture.

When you need a satellite

By default, all health checks run on the core Checkstack container. That works as long as:

• The targets you want to monitor are reachable from the core container’s network.
• Geographic latency between core and target is not part of what you are measuring.

Satellites come in when one of those breaks:

• Network isolation. The target is in a private network the core cannot reach (a customer VPC, a separate Kubernetes cluster, a network-segmented PCI environment). Deploy a satellite inside that network. The satellite opens an outbound WebSocket to the core; you do not need to expose the target to the core or punch firewall holes inbound.
• Geo-distribution. You want to measure latency or availability from multiple regions. Deploy a satellite per region. Each satellite executes the check independently, and the UI shows you per-region results.
• Probe locality. Some checks (large SSH command outputs, heavy SQL queries) are cheaper to run close to the target. A satellite in the same region as the target keeps that cost off your main link.

Note

A satellite is not a fallback or a high-availability mechanism. It is a way to extend reach. The core remains the single source of truth; satellites are stateless executors.

The pairing model

A satellite identifies itself with two pieces of credential material:

• A client ID (the satellite’s UUID, generated when an admin creates the satellite record in the UI).
• A token (a pre-shared API token, generated alongside the client ID and shown exactly once).

The pairing flow:

1. An admin clicks Add satellite in the Checkstack UI under Infrastructure -> Satellites. They give it a name and a region label.
2. Checkstack generates the client ID and a fresh token. The token is shown only once; copy it now or rotate later.
3. The admin deploys the satellite container with three environment variables:
   • CHECKSTACK_CORE_URL: the URL of the core Checkstack instance.
   • CHECKSTACK_SATELLITE_CLIENT_ID: the satellite’s UUID from step 2.
   • CHECKSTACK_SATELLITE_TOKEN: the API token from step 2.
4. The satellite container starts, opens a WebSocket to the core, authenticates with its client ID and token, and starts heartbeating.
5. The core marks the satellite online and starts sending it check execution jobs.

Tokens are stored as bcrypt hashes server-side. If a token is lost, you cannot recover it. Rotate the token from the satellite detail page; the rotation invalidates the old token and shows a fresh one.

Assigning checks to satellites

By default a health check assignment runs locally on the core. To use satellites, edit the assignment on a system’s detail page:

• Satellite IDs. Pick one or more satellites to run the check from.
• Include local. When satellites are selected, you can decide whether the core also continues running the check locally in parallel. Default is on.
• Environments per satellite. Once a satellite is assigned, you can scope it to specific environments. The default is all of them.

Each result is tagged with its source: null for local execution, the satellite’s UUID otherwise. The UI shows a human-readable source label (for example, “EU West (eu-west-1)”) on each run row and aggregates per-source for charts.

Tip

Running both local and satellite execution in parallel is the typical pattern when adding a satellite to an existing check. You see the local result you already trusted alongside the new per-region result; once the satellite is proven you can toggle local off.

Satellites and environments

A satellite fans a check out exactly as the core does: if the assignment covers three environments, the satellite runs the check three times, once per environment, and reports each result against that environment. Per-environment history, charts and rollups therefore include satellite results, and collectors running on a satellite get the same {{ environment.<key> }} templating they get locally.

Scope each satellite to the environments it can actually reach. A satellite inside the staging network has no route to production, so probing it there produces failures that say nothing about production:

• All environments (the default) - the satellite runs every environment the assignment covers, and automatically picks up new ones.
• Specific environments - the satellite runs only the ones you tick.

A satellite can only ever narrow the assignment’s own environment selector, never widen it: the assignment decides which environments the check covers, and the satellite decides which of those it is responsible for. Ticking nothing opts that satellite out of environment fan-out entirely - it runs the check once, with no environment in context.

Note

A satellite older than this feature is sent no environments and keeps reporting env-less results, exactly as it did before. Upgrade the satellite to get per-environment attribution.

How health is decided across locations

Every (environment, location) pair is evaluated independently against the check’s thresholds, and the worst result decides the check’s status. A check that passes from the core but fails from a satellite is unhealthy, not healthy - the service is unreachable for whoever that satellite speaks for.

The system overview names the location on each row once a check runs from more than one place, so you can see which one is failing without opening run history. The dashboard’s “X of Y checks failing” counts these slices, so a check probing one environment from the core and one satellite counts as two.

Do not assign a satellite to a check it has no route to. A satellite that fails every run makes the check unhealthy - which is correct, and is the whole point - so scope satellites to the environments they can actually reach (above) rather than leaving a permanently-failing probe in place.

Retiring a location retires its verdict with it: remove a satellite from the check (or turn Include local off) and its slice stops counting immediately. Its history is preserved under Old checks in the system overview.

Forwarding telemetry and scraping metrics

Beyond executing health checks, a satellite can act as a telemetry relay for the network zone it lives in. Because the satellite already holds one authenticated, outbound WebSocket to the core, it lets shippers and exporters inside the zone reach Checkstack without punching an inbound firewall hole to the core. The satellite is the single outbound connection; everything inside the zone talks to the satellite, and the satellite forwards to the core.

There are two shapes to this:

• Receive and forward. The satellite runs local receivers (OTLP and native HTTP for logs and metrics, plus an optional RFC 5424 syslog listener). A shipper inside the zone points at the satellite instead of at the core, and the satellite forwards the received telemetry over its WebSocket channel. This is the push model, moved one hop closer to the source.
• Pull and forward. The core cannot reach a target inside the zone (a Prometheus exporter, a Kubernetes API server), but the satellite can. Bind a pull telemetry source to a satellite, and the satellite runs it on its interval and forwards the records. This is the pull model, executed from inside the zone.

A satellite advertises which of these it can do as capabilities, shown as badges on its detail page: telemetry (the forwarding channel is enabled), log-receivers (the HTTP log and metric receivers are listening), syslog (the syslog receiver is listening), and telemetry-pull (satellite-side execution of bound pull sources is enabled). Capabilities come from the satellite’s environment configuration; see Connect a satellite for the flags.

How forwarded telemetry is authorized

The two shapes are authorized differently, because they carry different proof of who is allowed to write:

• Receiver forwarding is authorized by the stream token. A shipper hands the satellite the same per-stream source token it would send to the HTTP push endpoint (ckls_ for a log stream, ckms_ for a metric stream). The satellite forwards that token unchanged, and the core verifies it exactly as it verifies the direct HTTP push, honoring revocation. The satellite is a relay, not a new trust boundary; it never mints authority of its own.
• Pull execution is authorized by the source binding. A pull source is bound to a specific satellite in the UI. The core accepts forwarded records only for a source whose bound satellite matches the satellite that sent them, so a satellite cannot forward records for a source it was never bound to. Binding a source to a satellite requires read access to that satellite and that the satellite advertise telemetry-pull.

Note

Secrets for authenticated pull sources (an exporter bearer, a Kubernetes API token) are never stored on the satellite and never travel in the configuration pushed to it. The core delivers each secret field just in time over the secure channel for every run, and the satellite holds it in memory only for that run. The pushed config carries only the names of the secret fields.

Reliability and dropped telemetry

The satellite buffers telemetry in bounded, in-memory buffers that drop the oldest items when full, and a credit window with per-batch acknowledgements paces delivery to the core. If a satellite disconnects, buffered items may be dropped rather than held indefinitely. The count of items dropped this way is surfaced as Dropped in transit on the log-stream and metric-stream overview pages, so a gap caused by a satellite outage is visible rather than silent.

Heartbeats and online status

The satellite emits a heartbeat on its WebSocket connection. The core keeps a last_heartbeat_at per satellite. A satellite is considered online while its connection is open; if the connection drops or stops heartbeating, it goes offline and the core stops queuing jobs to it.

What happens to the checks it was running

A check assigned only to satellites (Include local off) is executed by those satellites and not by the core. If every satellite assigned to it is offline, nobody runs it - so the core records a degraded run carrying a “no assigned satellite is online” message.

Degraded, not unhealthy: the target may be perfectly fine, and what actually failed is our ability to observe it. Marking it unhealthy would raise incident-grade alarms about healthy services every time a satellite host reboots.

This is why the state matters. Recording nothing at all - which is what happened before - left the check displaying its last known status indefinitely, so a probe that had stopped running looked exactly like one that was passing. If a check’s satellites are all down, you should see that, not a stale green.

Checks also surface how old their last run is. When a check has been silent for five intervals (and at least ten minutes), its last run stat is highlighted and labelled stale, so an ageing status is visible even when no run was recorded to explain it. A paused check is never stale, and neither is a retired slice - one whose environment was removed from the system, or whose satellite was unassigned - because it stopped on purpose.

Note

These degraded runs do not send a notification each. One satellite going offline degrades every check assigned to it at once, so alerting per check would bury the actual cause under its own symptoms. The satellite’s Satellite connectivity subscription reports the offline satellite once; the degraded runs are still recorded, so the dashboard and the check’s history stay accurate.

The satellites list in Infrastructure -> Satellites shows current online state, last heartbeat timestamp, satellite version, and tags.

How long before a satellite counts as offline

By default a satellite is reported offline once its heartbeat is 45 seconds old (three heartbeat intervals). That is right for a satellite on a reliable link and too twitchy for one on a metered or intermittent uplink, so the tolerance is per satellite: edit it and pick an Offline after value, from 2 minutes up to 24 hours, or leave it on the platform default.

The value is a property of the link, not of the platform. Raising it for one flaky satellite does not make every other satellite slower to report.

Note

The same threshold governs every reader - the satellites list, the automation entity, and the background heartbeat monitor - so they can never disagree about whether a given satellite is online.

Being told when a satellite goes offline

A satellite going quiet is consequential and easy to miss: the checks it executes simply stop producing runs, so the systems it probes keep displaying their last known status.

Subscribe to it directly. Under Notification settings, each satellite offers a Satellite connectivity subscription that notifies you when it stops heartbeating (a warning) and when it comes back (informational). Notifications are collapsed per satellite, so a flapping link replaces its own previous notice instead of stacking one per transition.

If you want different routing or richer conditions, the same transitions are also available as automation triggers - satellite.connected, satellite.disconnected, and satellite.heartbeat_lost - which you can wire to any automation action. Use a subscription for “tell me”, and an automation for “do something”.

Tags

Satellites carry a free-form tags map (key/value strings). Use tags to organise satellites by environment, cloud provider, customer tenant, or anything else that matters in your setup. Tags are advisory metadata today; they do not yet drive automatic check assignment.

Versioning and updates

Satellites report their version on connect. The core does not auto-update satellites; you upgrade them the same way you upgrade the core: pull a new image, restart the container. Keep your satellite version close to your core version; very stale satellites may not understand newer strategies.

Older satellites may lack support for strategies introduced after their build. If you install a new health check strategy plugin on the core and assign it to an old satellite, the satellite will reject the job. Upgrade the satellite or pick a newer one.

Security model

• The satellite connection is outbound from the satellite to the core. You do not have to expose the satellite to the internet.
• The token authenticates the satellite to the core, proving WHICH satellite is connected.
• The core also authorizes WHAT a satellite may report for: a result message is accepted only when its (configId, systemId) pair is in that satellite’s current assignment set. A satellite cannot forge results for a system it is not assigned. The assignment set is the durable source of truth and is re-read on every assignment change, so a reassignment takes effect immediately. An out-of-scope result is logged and dropped without tearing down the connection.
• Config relayed to the satellite (credentials in the check config, for example) is sent over the authenticated connection. The satellite uses the relayed config only for the duration of the run and does not persist it.

Danger

A satellite has the same secret material reach as the core for the checks assigned to it. Treat satellite hosts as carefully as you would the core: hardened image, restricted SSH, monitored.

UI tour

Where to go	What you do there
Infrastructure -> Satellites	List, create, delete, and rotate tokens for satellites.
Satellite detail	See online status, last heartbeat, version, tags, and capability badges. Rotate the token.
System detail -> Health check assignment	Pick which satellites execute the check. Toggle Include local.
Health check run row	See the source label per result (Local, EU West, …).

Where to go next

• Hands-on. Walk through Connect a satellite.
• Per-region checks. See Health checks for how satellite-tagged runs feed into aggregates.
• GitOps. GitOps can declare satellite records and tags as YAML.