Workers
A worker is a service of kind worker: a long-running process with no port, no domain and no public check. It is built
and released with its environment like any other service (see Environments & Services), and Canopy runs
it, restarts it when it crashes and shows how it is doing.
Adding a worker as a separate service uses host.environment_services, which is generally available and enabled by
default, on a Forest environment. You can also create an environment whose default service is a worker. Native Groves
can run a worker with one replica, but require it to stay at one and do not support a grace period; scale or pause
settings that change the replica count, and any configured grace period, are rejected for native workloads. See
Native Groves for runtime-specific behavior.
Create one
Section titled “Create one”canopy workers create --env <environment_id> --name consumer --start-command "node worker.js"An environment can also start as a worker: canopy environments create <branch> --project <slug> --repository <owner/repo> --kind worker. A worker has one process, always named worker. When the repository’s .canopy.yml declares
processes, the worker takes the entry named like the service, else the entry named worker; a web entry is never
used. The service’s own settings override that entry.
How it is doing
Section titled “How it is doing”canopy workers status consumer --env <environment_id>The status is one word, computed by Canopy and the same in the CLI, the API, MCP and the app:
| Status | Meaning |
|---|---|
running |
Every process is ready and the Grove reported in the last five minutes |
starting |
A rollout is short of ready pods |
degraded |
Some pods are not ready and no rollout is running |
crash_looping |
A pod keeps failing and restarting, or cannot pull its image |
stopped |
Scaled to zero replicas |
unknown |
The Grove has not reported for more than five minutes; never shown as running |
not_deployed |
No release has been deployed |
idle |
A job, which runs on demand |
The service page in the app shows the same, with each pod, the last termination (“Out of memory”, “Exited 1”) and the problems Canopy found.
Logs, including the crash
Section titled “Logs, including the crash”canopy workers logs consumer --env <environment_id> [--tail] [--lines 200]canopy workers logs consumer --env <environment_id> --previousThe live log includes pods that are running but not ready, so a worker without a readiness check can still be read.
--previous returns the complete log of the container that was restarted last, which is where the reason for a crash is.
It needs a Grove running a current controller; an older one answers with the current log.
Stop, start, scale, restart
Section titled “Stop, start, scale, restart”canopy workers pause consumer --env <environment_id>canopy workers resume consumer --env <environment_id> [--replicas 2]canopy workers scale consumer --env <environment_id> --replicas 3canopy workers restart consumer --env <environment_id>restart rolls every pod from the image it already runs with the same configuration, even when nothing changed. It needs
the environment:restart permission, which developers have, like scaling.
Settings
Section titled “Settings”| Setting | Flag | Meaning |
|---|---|---|
| Grace period | --grace-period <seconds> |
How long a process gets after SIGTERM before it is killed (1 to 3600, default 30). Handle SIGTERM: finish or hand back the job you are on, then exit. |
| Rollout | --rollout rolling|recreate |
recreate stops every pod before the new ones start, so one instance never runs twice and no spare capacity is needed. It is the default for a worker. rolling starts the new pod first. |
| Liveness | --liveness-command <cmd> |
A command run inside the container every 30 seconds after a 60 second start-up grace. Three failures in a row restart the container. |
| Readiness | --readiness-probe-command <cmd> |
A command run inside the container every 30 seconds. While it fails the pod counts as not ready. |
A worker has no port to probe, so these commands are how you tell Canopy it is healthy. A pattern that works in any language: the worker touches a file at the end of every loop, and the liveness command checks the file is recent:
canopy app-services update consumer --env <environment_id> \ --liveness-command 'test $(( $(date +%s) - $(stat -c %Y /tmp/alive) )) -lt 120'Findings, alerts and what “healthy” means
Section titled “Findings, alerts and what “healthy” means”Canopy checks every worker and web service once a minute against what its Grove last reported, and keeps one open finding per service and kind until the problem has been gone for two checks in a row. A crash loop that flickers between running and failing stays one finding.
| Finding | Opens when |
|---|---|
crash_looping |
A pod keeps failing and restarting |
oom |
A container was killed for running out of memory in the last 15 minutes |
unavailable |
No pod is ready (after the service had 3 minutes to settle), or its image cannot be pulled |
missing |
The Grove reports no process for the service |
liveness_failing |
The liveness command keeps failing |
status_stale |
The Grove has not reported for more than 5 minutes (the message says whether the Grove itself is answering) |
A job is never judged. While a finding is open, a successful deploy no longer reads healthy: a crash loop, no ready pod
or a missing service makes the environment unhealthy, an out-of-memory kill or a failing liveness check degraded, and
a report that stopped unknown, never healthy. canopy workers status lists the findings, the service page shows
them, and GET /environments/{id}/state (its health section) carries them.
Every finding that opens or resolves is a platform event (workload.unhealthy, workload.recovered). Each becomes a
notification for the team with a link to the service page, and a webhook subscription can listen for it. For alert rules
there is a metric, unhealthy_workloads_count: the number of services in the team with an open finding.
Telling Canopy the application is alive
Section titled “Telling Canopy the application is alive”A process can be running and still stuck. The pattern that works in any language: the worker touches a file at the end of each loop, and the liveness command checks the file is recent (see Settings above). Examples for the file:
import { utimesSync, writeFileSync, existsSync } from 'node:fs';const beat = () => (existsSync('/tmp/alive') ? utimesSync('/tmp/alive', new Date(), new Date()) : writeFileSync('/tmp/alive', ''));# Pythonfrom pathlib import Pathdef beat(): Path("/tmp/alive").touch()// Gofunc beat() { f, _ := os.Create("/tmp/alive"); f.Close() }void Beat() => File.WriteAllText("/tmp/alive", DateTimeOffset.UtcNow.ToString("O"));Call it once per loop, not from a separate timer thread, so a stuck loop stops the heartbeat.
One-off commands
Section titled “One-off commands”canopy run --env <environment_id> --service consumer -- node migrate.jscanopy jobs list --env <environment_id> --kind run --service consumerA command runs in the service’s image and its own process.
Sizing
Section titled “Sizing”Each worker reserves its CPU and memory times its replicas on its Grove (canopy app-services footprint). Set
--cpu-limit and --memory-limit on the service; they override the numbers in .canopy.yml.