Skip to content

Workers

A worker is a service of kind worker: a long-running process with no port, no domain and no public check. It is built and released with its environment like any other service (see Environments & Services), and Canopy runs it, restarts it when it crashes and shows how it is doing.

Adding a worker as a separate service uses host.environment_services, which is generally available and enabled by default, on a Forest environment. You can also create an environment whose default service is a worker. Native Groves can run a worker with one replica, but require it to stay at one and do not support a grace period; scale or pause settings that change the replica count, and any configured grace period, are rejected for native workloads. See Native Groves for runtime-specific behavior.

canopy workers create --env <environment_id> --name consumer --start-command "node worker.js"

An environment can also start as a worker: canopy environments create <branch> --project <slug> --repository <owner/repo> --kind worker. A worker has one process, always named worker. When the repository’s .canopy.yml declares processes, the worker takes the entry named like the service, else the entry named worker; a web entry is never used. The service’s own settings override that entry.

canopy workers status consumer --env <environment_id>

The status is one word, computed by Canopy and the same in the CLI, the API, MCP and the app:

Status Meaning
running Every process is ready and the Grove reported in the last five minutes
starting A rollout is short of ready pods
degraded Some pods are not ready and no rollout is running
crash_looping A pod keeps failing and restarting, or cannot pull its image
stopped Scaled to zero replicas
unknown The Grove has not reported for more than five minutes; never shown as running
not_deployed No release has been deployed
idle A job, which runs on demand

The service page in the app shows the same, with each pod, the last termination (“Out of memory”, “Exited 1”) and the problems Canopy found.

canopy workers logs consumer --env <environment_id> [--tail] [--lines 200]
canopy workers logs consumer --env <environment_id> --previous

The live log includes pods that are running but not ready, so a worker without a readiness check can still be read. --previous returns the complete log of the container that was restarted last, which is where the reason for a crash is. It needs a Grove running a current controller; an older one answers with the current log.

canopy workers pause consumer --env <environment_id>
canopy workers resume consumer --env <environment_id> [--replicas 2]
canopy workers scale consumer --env <environment_id> --replicas 3
canopy workers restart consumer --env <environment_id>

restart rolls every pod from the image it already runs with the same configuration, even when nothing changed. It needs the environment:restart permission, which developers have, like scaling.

Setting Flag Meaning
Grace period --grace-period <seconds> How long a process gets after SIGTERM before it is killed (1 to 3600, default 30). Handle SIGTERM: finish or hand back the job you are on, then exit.
Rollout --rollout rolling|recreate recreate stops every pod before the new ones start, so one instance never runs twice and no spare capacity is needed. It is the default for a worker. rolling starts the new pod first.
Liveness --liveness-command <cmd> A command run inside the container every 30 seconds after a 60 second start-up grace. Three failures in a row restart the container.
Readiness --readiness-probe-command <cmd> A command run inside the container every 30 seconds. While it fails the pod counts as not ready.

A worker has no port to probe, so these commands are how you tell Canopy it is healthy. A pattern that works in any language: the worker touches a file at the end of every loop, and the liveness command checks the file is recent:

canopy app-services update consumer --env <environment_id> \
--liveness-command 'test $(( $(date +%s) - $(stat -c %Y /tmp/alive) )) -lt 120'

Findings, alerts and what “healthy” means

Section titled “Findings, alerts and what “healthy” means”

Canopy checks every worker and web service once a minute against what its Grove last reported, and keeps one open finding per service and kind until the problem has been gone for two checks in a row. A crash loop that flickers between running and failing stays one finding.

Finding Opens when
crash_looping A pod keeps failing and restarting
oom A container was killed for running out of memory in the last 15 minutes
unavailable No pod is ready (after the service had 3 minutes to settle), or its image cannot be pulled
missing The Grove reports no process for the service
liveness_failing The liveness command keeps failing
status_stale The Grove has not reported for more than 5 minutes (the message says whether the Grove itself is answering)

A job is never judged. While a finding is open, a successful deploy no longer reads healthy: a crash loop, no ready pod or a missing service makes the environment unhealthy, an out-of-memory kill or a failing liveness check degraded, and a report that stopped unknown, never healthy. canopy workers status lists the findings, the service page shows them, and GET /environments/{id}/state (its health section) carries them.

Every finding that opens or resolves is a platform event (workload.unhealthy, workload.recovered). Each becomes a notification for the team with a link to the service page, and a webhook subscription can listen for it. For alert rules there is a metric, unhealthy_workloads_count: the number of services in the team with an open finding.

A process can be running and still stuck. The pattern that works in any language: the worker touches a file at the end of each loop, and the liveness command checks the file is recent (see Settings above). Examples for the file:

Node.js
import { utimesSync, writeFileSync, existsSync } from 'node:fs';
const beat = () => (existsSync('/tmp/alive') ? utimesSync('/tmp/alive', new Date(), new Date()) : writeFileSync('/tmp/alive', ''));
# Python
from pathlib import Path
def beat(): Path("/tmp/alive").touch()
// Go
func beat() { f, _ := os.Create("/tmp/alive"); f.Close() }
.NET
void Beat() => File.WriteAllText("/tmp/alive", DateTimeOffset.UtcNow.ToString("O"));

Call it once per loop, not from a separate timer thread, so a stuck loop stops the heartbeat.

canopy run --env <environment_id> --service consumer -- node migrate.js
canopy jobs list --env <environment_id> --kind run --service consumer

A command runs in the service’s image and its own process.

Each worker reserves its CPU and memory times its replicas on its Grove (canopy app-services footprint). Set --cpu-limit and --memory-limit on the service; they override the numbers in .canopy.yml.