Skip to content

Run container tasks

An execution backend runs container tasks. A model binding describes the work of a scoring run as a plan of such tasks, ContainerTaskSpec values, and the same task runs on local Docker, on Modal or on Kubernetes. run_scoring and altar run drive a backend for you. Drive one yourself when your own code submits and tracks tasks. To pick a backend in a run spec instead, see Run tasks on a GPU or a cluster.

The names on this page come from altar.execution and altar.models and are stable.

Choose a backend

altar.execution has three backends: LocalDockerExecutionBackend (registry name local, Stable), ModalExecutionBackend (modal, Preview) and KubernetesExecutionBackend (kubernetes, Preview). Platform support lists what each needs and what it is tested on. Each backend handles images, staging, polling and capacity reporting itself.

from altar.execution import LocalDockerExecutionBackend

backend = LocalDockerExecutionBackend(data_root="/scratch/altar-volume")
resolver = backend.path_resolver()  # give this to the ScoringRequest

Drive a plan's tasks

A driver runs a plan in these steps:

stage declared inputs
        ↓
submit independent tasks
        ↓
poll task handles
        ↓
submit a dependent task when its prerequisites succeed
        ↓
collect declared outputs

Your program makes each step's call: stage_inputs, submit, poll, cancel and collect_outputs. run_scoring and run_model_workflow make these calls for you.

Map logical paths to files

A task's command reads and writes paths under one container mount root. A PathResolver turns a logical path, such as jobs/demo/variants.tsv, into that command path. Transfer values tell the backend which files to stage in before a task and collect after it, and may carry a digest the task checks.

A Kubernetes backend on a shared volume and a backend that stages files for each task run the same plan. Only their transfer implementations differ.

Track task state

submit returns a TaskHandle. poll returns a TaskStatus for each handle, with a TaskState such as pending, running, succeeded or failed. Record handles in a TaskLedger when a driver must survive a restart. InMemoryTaskLedger keeps them in memory; an application with a database can implement its own ledger.

Some plans have a dependent step, for example scoring several folds and then summarizing them. GroupSpec says when a group of tasks is ready for its dependent step, and reconcile_once submits that step exactly once. run_reconcile_loop repeats it until the ledger is idle, or until max_ticks checks or timeout_s seconds have passed, and hands each check's ReconcileResult, with its counts of running and pending tasks, to on_tick=. Scoring through run_scoring needs neither.

Run a workflow plan

Model preparation and interpretation use a ModelWorkflowPlan: a graph of named WorkflowStage values. altar.models.run_model_workflow runs one. It stages external inputs, submits every stage whose inputs are ready, polls until those stages finish, and collects the declared outputs. It waits at most timeout_s seconds (24 hours by default) for each wave of stages, checking every poll_interval_s seconds, and passes progress lines to progress=, as run_scoring does; see Container plans. To save a plan before running it, use workflow_plan_to_json and workflow_plan_from_json. The plan holds no client or credential, so a scheduler of your own can store it and run it later.

Scoring uses its own plan types, InlineScoringPlan and ContainerScoringPlan, which carry batch packing, result decoding and store writes.

Handle failed tasks

A failed task's TaskStatus.failure is a TaskFailure. Its reason is a stable code, the same whichever backend ran the task, so your code can branch on it:

Reason Meaning
exit_<n> The command exited with code n (also in exit_code).
timeout The task ran longer than its timeout_s.
oom_killed The platform killed the task for exceeding its memory limit.
input_digest_mismatch A content-addressed input was missing or did not match its digest.
output_clear_failed A file an earlier run left at an output could not be deleted before the task started.
container_missing The backend can no longer find the task, for example because it was deleted.
job_failed The platform reported failure without a finer reason.

detail holds the platform's message for a person to read. deterministic is true when a retry would fail the same way, such as a nonzero exit, and false for a transient infrastructure failure.

Retry policy, stalled-task detection and platform-specific fixes are yours to write. No reference backend implements recover or honors avoid_hosts.

To retry, submit the same specs again; no backend rejects a spec it has seen. The local and Modal backends launch a new task. The Kubernetes backend names each Job by a hash of its spec. It returns the existing Job while that Job is pending or running, and deletes and recreates it once it has finished, whether it succeeded or failed. A later run under the same job_id therefore computes its own results. It raises instead of adopting an active Job that a differently configured backend created (see Accept resubmission).

Limits every backend enforces

Every reference backend enforces a task's timeout_s. A task without one has no limit on local Docker and Kubernetes. A Modal sandbox lives at most 24 hours, so such a task gets 24 hours on Modal, and a timeout_s above 24 hours is rejected when the task is submitted.

Every reference backend checks content-addressed inputs before the task's command runs. A verified file gets a verification record beside it, so a later task that reads the same unchanged file checks its metadata instead of hashing it again (see Hash once per volume).

A task that requests GPUs gets exactly the number of devices it asks for, or the backend fails it; a backend never runs a GPU task on the CPU instead. What a runtime does inside its container is its own choice: the Cherimoya and Sei runtimes use the CPU when no GPU is visible.

Next