Run container tasks¶
An execution backend runs container tasks. A model binding describes the work of a scoring
run as a plan of such tasks, ContainerTaskSpec values, and the same task runs on local Docker, on Modal or on
Kubernetes. run_scoring and altar run drive a backend for you. Drive one yourself when your own code submits and
tracks tasks. To pick a backend in a run spec instead, see Run tasks on a GPU or a cluster.
The names on this page come from altar.execution and altar.models and are stable.
Choose a backend¶
altar.execution has three backends: LocalDockerExecutionBackend (registry name local, Stable),
ModalExecutionBackend (modal, Preview) and KubernetesExecutionBackend (kubernetes, Preview).
Platform support lists what each needs and what it is tested on. Each
backend handles images, staging, polling and capacity reporting itself.
from altar.execution import LocalDockerExecutionBackend
backend = LocalDockerExecutionBackend(data_root="/scratch/altar-volume")
resolver = backend.path_resolver() # give this to the ScoringRequest
Drive a plan's tasks¶
A driver runs a plan in these steps:
stage declared inputs
↓
submit independent tasks
↓
poll task handles
↓
submit a dependent task when its prerequisites succeed
↓
collect declared outputs
Your program makes each step's call: stage_inputs, submit, poll, cancel and collect_outputs. run_scoring
and run_model_workflow make these calls for you.
Map logical paths to files¶
A task's command reads and writes paths under one container mount root. A PathResolver turns a logical path, such
as jobs/demo/variants.tsv, into that command path. Transfer values tell the backend which files to stage in before
a task and collect after it, and may carry a digest the task checks.
A Kubernetes backend on a shared volume and a backend that stages files for each task run the same plan. Only their transfer implementations differ.
Track task state¶
submit returns a TaskHandle. poll returns a TaskStatus for each handle, with a TaskState such as pending,
running, succeeded or failed. Record handles in a TaskLedger when a driver must survive a restart.
InMemoryTaskLedger keeps them in memory; an application with a database can implement its own ledger.
Some plans have a dependent step, for example scoring several folds and then summarizing them. GroupSpec says when a
group of tasks is ready for its dependent step, and reconcile_once submits that step exactly once.
run_reconcile_loop repeats it until the ledger is idle, or until max_ticks checks or timeout_s seconds have
passed, and hands each check's ReconcileResult, with its counts of running and pending tasks, to on_tick=. Scoring
through run_scoring needs neither.
Run a workflow plan¶
Model preparation and interpretation use a ModelWorkflowPlan: a graph of named WorkflowStage values.
altar.models.run_model_workflow runs one. It stages external inputs, submits every stage whose inputs are ready, polls
until those stages finish, and collects the declared outputs. It waits at most timeout_s seconds (24 hours by
default) for each wave of stages, checking every poll_interval_s seconds, and passes progress lines to progress=,
as run_scoring does; see Container plans. To save a plan before running it, use
workflow_plan_to_json and workflow_plan_from_json. The plan holds no client or credential, so a scheduler of your
own can store it and run it later.
Scoring uses its own plan types, InlineScoringPlan and ContainerScoringPlan, which carry batch packing, result
decoding and store writes.
Handle failed tasks¶
A failed task's TaskStatus.failure is a TaskFailure. Its reason is a stable code, the same whichever backend ran
the task, so your code can branch on it:
| Reason | Meaning |
|---|---|
exit_<n> |
The command exited with code n (also in exit_code). |
timeout |
The task ran longer than its timeout_s. |
oom_killed |
The platform killed the task for exceeding its memory limit. |
input_digest_mismatch |
A content-addressed input was missing or did not match its digest. |
output_clear_failed |
A file an earlier run left at an output could not be deleted before the task started. |
container_missing |
The backend can no longer find the task, for example because it was deleted. |
job_failed |
The platform reported failure without a finer reason. |
detail holds the platform's message for a person to read. deterministic is true when a retry would fail the same
way, such as a nonzero exit, and false for a transient infrastructure failure.
Retry policy, stalled-task detection and platform-specific fixes are yours to write. No reference backend implements
recover or honors avoid_hosts.
To retry, submit the same specs again; no backend rejects a spec it has seen. The local and Modal backends launch a new
task. The Kubernetes backend names each Job by a hash of its spec. It returns the existing Job while that Job is pending
or running, and deletes and recreates it once it has finished, whether it succeeded or failed. A later run under the
same job_id therefore computes its own results. It raises instead of adopting an active Job that a differently
configured backend created (see Accept resubmission).
Limits every backend enforces¶
Every reference backend enforces a task's timeout_s. A task without one has no limit on local Docker and Kubernetes.
A Modal sandbox lives at most 24 hours, so such a task gets 24 hours on Modal, and a timeout_s above 24 hours is
rejected when the task is submitted.
Every reference backend checks content-addressed inputs before the task's command runs. A verified file gets a verification record beside it, so a later task that reads the same unchanged file checks its metadata instead of hashing it again (see Hash once per volume).
A task that requests GPUs gets exactly the number of devices it asks for, or the backend fails it; a backend never runs a GPU task on the CPU instead. What a runtime does inside its container is its own choice: the Cherimoya and Sei runtimes use the CPU when no GPU is visible.
Next¶
- Store and assemble results reads back what the tasks scored.
- Run tasks on a GPU or a cluster picks a backend in a run spec.
- Add an execution backend writes a backend for another platform.