Add an execution backend¶
An execution backend runs container tasks on a compute system: it submits each task and reports its state. A task is one unit of work, such as one model fold in one container. The backend runs the command a model binding built, unchanged, and never reads the model's scores.
When to choose an execution backend¶
Write one when your tasks must run somewhere the shipped backends don't reach, such as a lab's batch queue or
another cloud's container service. Altar ships local (Docker on one machine), kubernetes and modal;
Platform support says how each is tested. Every model binding's plan
runs on every backend, so a new backend needs no change to any binding. Choose an interface lists the
other extension points.
The interface¶
Subclass ExecutionBackend, imported from altar.execution. It is Stable.
| Member | Required? | What it is |
|---|---|---|
name |
required | The registry name, and the type of a run spec's [backend] section. |
submit(spec) |
required | Launches one ContainerTaskSpec and returns a TaskHandle that still works after the submitting process restarts. |
poll(handles) |
required | Returns one current TaskStatus per handle, in order, without blocking. |
input_digest_verification |
required: "staging" or "task" |
Where the backend checks a content-addressed input. The default, None, fails the conformance suite. |
stage_inputs(transfers), collect_outputs(transfers) |
default: do nothing | Copy inputs to where the task reads them, and outputs back to their URIs. |
path_resolver() |
default: paths unchanged | Turns a logical path into the path the task sees, such as /mnt/volume/jobs/.... |
failure_detail(handle) |
default None |
The last lines of a failed task's log. |
cancel(handle) |
default: do nothing | Stops or deletes a task. A missing task counts as cancelled. |
capacity() |
default: an empty report | Free slots per pool, for a scheduler of your own. Altar never reads it. |
gpu_pools() |
provisional; default None, any pool |
The GPU pool names the backend serves. altar run refuses a model whose pool is not one of them before anything is submitted; see Declare your GPU pools. |
await_completion(handles), recover(failures, banned=) |
default: not supported, do nothing | For a backend with a native wait, or one that can restart failed tasks. |
adopts_active_tasks, clears_outputs_on_fresh_start, writes_output_in_place(transfer) |
provisional; default False |
Who deletes outputs an earlier run left; see Clear outputs an earlier run left. |
A provisional member is expected to change; see Provisional names.
A minimal backend¶
This backend runs each task as a job on a lab's batch queue, through the queue's own client. The queue's workers
mount the lab's shared filesystem, so a task reads each input at its path, and stage_inputs checks the bytes
there. Put it in a module of your own, such as lab_queue_backend.py:
from typing import Any
from altar.execution import (
ContainerTaskSpec,
ExecutionBackend,
TaskFailure,
TaskHandle,
TaskState,
TaskStatus,
Transfer,
verify_transfer_digest,
)
class LabQueueBackend(ExecutionBackend):
"""Run each task as one job on the lab's batch queue, whose workers mount the lab's shared filesystem."""
name = "lab_queue"
input_digest_verification = "staging"
def __init__(self, client: Any) -> None:
self.client = client # the queue's client; the caller created it, so the caller closes it
async def stage_inputs(self, transfers: list[Transfer]) -> None:
# Workers read each input at its path on the shared filesystem, so check the bytes there.
for transfer in transfers:
verify_transfer_digest(transfer, transfer.uri)
async def submit(self, spec: ContainerTaskSpec) -> TaskHandle:
job_id = await self.client.launch(
image=spec.image,
command=spec.command,
gpu_type=spec.resources.gpu,
gpu_count=spec.resources.count if spec.resources.gpu else 0,
memory_gb=spec.resources.memory_gb,
timeout_s=spec.timeout_s,
labels=spec.labels,
)
return TaskHandle(
backend=self.name, external_id=job_id, labels=dict(spec.labels), plugin_identity=spec.plugin_identity
)
async def poll(self, handles: list[TaskHandle]) -> list[TaskStatus]:
return [self._status(handle, await self.client.describe(handle.external_id)) for handle in handles]
def _status(self, handle: TaskHandle, job: dict[str, Any] | None) -> TaskStatus:
if job is None:
failure = TaskFailure("container_missing", "the queue has no such job")
return TaskStatus(handle, TaskState.FAILED, failure)
if job["state"] in ("queued", "running"):
return TaskStatus(handle, TaskState.PENDING if job["state"] == "queued" else TaskState.RUNNING)
if job["state"] == "timed_out":
return TaskStatus(handle, TaskState.FAILED, TaskFailure("timeout", "the job ran past its time limit"))
code = job["exit_code"]
if code == 0:
return TaskStatus(handle, TaskState.SUCCEEDED)
failure = TaskFailure(f"exit_{code}", f"the job exited with code {code}", deterministic=True, exit_code=code)
return TaskStatus(handle, TaskState.FAILED, failure)
What each part does:
submitpasses the image, the command, the resource request, the time limit and the labels to the queue unchanged, and returns a handle whoseexternal_idis the queue's job ID.pollasks the queue about each handle and maps its state to an AltarTaskState. A job the queue no longer knows is a terminal failure with reasoncontainer_missing.stage_inputschecks each content-addressed input withverify_transfer_digest, which raisesTransferIntegrityErroron a mismatch, a missing file or a directory.
A queue whose workers can't see the submitting machine's files copies each input to storage the workers read in
stage_inputs, hashing the bytes it writes, and copies each output back in collect_outputs.
Honor the task declaration¶
The backend must map:
- image and command without model-specific rewrites;
- CPU, memory, and GPU requests, with
ResourceRequest.countas the exact number of devices; - logical input and output transfers;
- structured labels;
timeout_sas a wall-clock limit enforced by the provider or bypoll, andNoneas no limit. If the provider caps task lifetime or applies a default when no limit is passed, request its maximum forNoneand reject a longer explicit limit;- provider state into Altar task states.
Keep task labels unchanged: the code that drives a run groups tasks and decides readiness by them. If the provider cannot represent something the spec asks for, fail the submission. A task that asks for a GPU must never run on a CPU instead.
Verify content-addressed inputs¶
An input Transfer with a digest names exact bytes, such as model weights: the SHA-256 of one regular file.
A digest never covers a directory, so a digest-bearing input that resolves to a directory or a missing file is
rejected like a mismatch. Every backend verifies digests before the task's command runs, and declares where
with the class attribute input_digest_verification:
"staging":stage_inputshashes the bytes it copies and raisesTransferIntegrityErroron a mismatch, as the local and Modal backends do."task": each task checks its own inputs first and fails with reasoninput_digest_mismatch. Use this when inputs already sit on a shared volume the submitting process cannot read; the Kubernetes backend adds an init container that runsverify_files_script()on the mounted paths.
The conformance suite fails a backend that declares neither.
Hash once per volume¶
A large input such as a reference genome is read by every task of every job. Hashing it in full before each
task can take longer than the task itself, so a verified file carries a verification record: a one-line file
beside it, at its path plus .sha256-verified.
The line names the digest the bytes matched, and the file's size, whole-second modification and status-change times, and inode number when they matched. Rewriting, replacing, or truncating the file changes at least one of those values, so the record no longer matches and the next check hashes the file again. A record is only written once the file's status-change second has passed, and only if the file did not change while it was hashed.
verify_file_digest(path, digest)accepts a file from a current record, or hashes it and writes one.verify_transfer_digest(transfer, path, record=True)always hashes, then writes a record. Use it where bytes enter storage your tasks read, and only for copies the backend owns.verify_files_script()is the same check as a POSIX shell program, for BusyBox or GNU coreutils.
The local backend records each verified copy and skips copying an input whose destination already holds a recorded copy of the same digest. The Kubernetes init container accepts recorded inputs and records the ones it hashes. Model runtimes check their inputs the same way, so once a resource is staged and recorded, each fold task reads only its metadata.
Anyone who can write the volume can also write a record. Where untrusted parties can write it, construct the
Kubernetes backend with trust_verification_records=False, which hashes every input in every task. Its init
container still writes a record after each successful hash, so a runtime that trusts records then accepts the
file from the record its own task's verifier just wrote instead of hashing it a second time.
Report task state¶
Use the reason codes documented on TaskFailure for the same events on every backend: exit_<n> with
exit_code for an application exit, timeout, oom_killed, input_digest_mismatch, container_missing
for a task the provider can no longer find, and job_failed when the provider gives no finer detail. A
missing task must be a terminal failure, never a task that stays running forever.
TaskHandle.external_id must be enough to poll the provider after the submitting process restarts. Use the
provider's own task ID, not a display name, and keep grouping information in the labels.
Accept resubmission¶
A failed or interrupted run is retried by submitting the same specs again, so submit must never raise
because a spec was submitted before. A backend whose provider mints a fresh ID per task, as Docker and Modal
do, launches another task. A backend that names tasks by their spec must settle the name conflict
itself:
- a task that is still pending or running is adopted:
submitreturns a handle to it, so the same work never runs twice at once; - a task that has finished, whether it succeeded or failed, is deleted and submitted again. A spec names its inputs by path, and a new run under the same job can put different content at the same path (a new variant batch, for example), so a finished task's output is never taken as the new run's.
How the Kubernetes backend resubmits¶
The Kubernetes backend names each Job by a hash of its spec and follows these rules:
- Its runtime reports a taken name as
K8sJobExistsError(the API server's409 AlreadyExists) and reads the existing Job withread_job. A customKubernetesRuntimemust raiseK8sJobExistsErrorfor a name conflict and report each Job's annotations, and should report its uid. - Each Job records a hash of its fully rendered manifest in the
altar.kundajelab.org/manifest-sha256annotation. The name covers only the spec, so an active Job whose manifest differs, for example one created by a backend with another PVC, verifier image, or node selector, is neither adopted nor deleted:submitraisesK8sJobExistsError. Let it finish, or delete it, before resubmitting. - An active Job without that annotation is refused the same way. A custom
KubernetesRuntimemust return annotations fromlist_jobsandread_job. - A finished Job is deleted with its pods before it is created again. The delete is conditioned on the Job's uid, so when two drivers resubmit at once, neither deletes the fresh Job the other just created; the slower one adopts it instead.
- If the existing Job disappears between the conflict and the read, or is still being deleted, the create is
retried with backoff for about a minute before
submitraisesK8sJobExistsError.
An adopted Job runs with whatever its input paths hold. Content-addressed inputs (a Transfer.digest) are
verified before it starts, but inputs named only by path are not. So if a driver crashes and a new run writes
different content to the same path before the old Job reads it, the adopted Job reads the new content.
Declare your GPU pools¶
ResourceRequest.gpu names a pool, such as the shipped models' gpu, and a spec's
[models.<name>] pool picks another one. A backend whose pools come from its own
options returns their names from gpu_pools(). altar run, altar prepare and altar interpret then check every
model's pools against them and stop, naming the pools the backend defines, before any task is submitted. Call
require_gpu_pool(self, spec.resources.gpu) at the top of submit so a program that submits directly gets the
same UnknownPoolError:
from altar.execution import require_gpu_pool
class LabQueueBackend(ExecutionBackend):
...
def gpu_pools(self) -> frozenset[str]:
return frozenset(self.client.gpu_types()) # the queue's own names, such as "gpu" and "a100"
async def submit(self, spec: ContainerTaskSpec) -> TaskHandle:
require_gpu_pool(self, spec.resources.gpu)
...
The default, None, accepts every name, so a backend that does not override it keeps its own handling of an
unknown pool. The conformance suite checks that a backend declaring its pools refuses any other at submit.
Clear outputs an earlier run left¶
A task must never hand back a file an earlier run left at one of its outputs. Two class attributes on
ExecutionBackend say who deletes those files:
adopts_active_tasks:submitmay return a handle to a task that is still pending or running instead of launching a new one. The default isFalse.clears_outputs_on_fresh_start:submitdeletes the file at each ofspec.outputsbefore it launches a fresh attempt, and keeps the outputs of a task it adopts. The default isFalse.
Both are provisional. A backend that adopts also says, through the
provisional method writes_output_in_place(transfer), whether its task writes an output's uri directly. The
default answer is False: collect_outputs copies the output there after the task succeeds, so a driver can
delete that location before submitting without losing an adopted task's work.
run_scoring reads them before it submits anything:
adopts_active_tasks |
clears_outputs_on_fresh_start |
What run_scoring does |
|---|---|---|
False |
either | Deletes each terminal output's file through its Storage, then submits. |
True |
True |
Deletes each terminal output that writes_output_in_place rejects, and leaves the rest to the backend. A restarted driver collects what an adopted task wrote. |
True |
False |
Raises ValueError before anything is submitted. |
A backend whose provider launches a new task on every submit, as Docker and Modal do, needs neither attribute;
the reference backends still declare clears_outputs_on_fresh_start, because their submit deletes the
outputs on the backend's own volume. A backend that adopts declares both, and clears only when it commits to a
new attempt. The Kubernetes backend does it inside the new Job: a clear-outputs init container deletes the
files at the outputs' mounted paths before the task starts, so an adopted Job, which starts no new pod, keeps
what it wrote. In the staged mode it also deletes them through its Storage before it creates the Job. A
failure of that init container is reported with reason output_clear_failed.
The Kubernetes backend writes an output in place when its uri is the output's volume location (staged mode),
or the path the pod writes, <mount_root>/<logical_path> (pre-staged mode). In the pre-staged mode nothing copies
an output anywhere else, so run_scoring deletes a file an earlier run left at any other location and the run
fails as not produced. A driver that sees the volume at a path other than mount_root should use the staged mode,
with volume_uri set to that path: outputs already at their volume location are not copied.
The clearing lives in submit, so a subclass that overrides submit reads clears_outputs_on_fresh_start as
False until its own class body sets it again. A KubernetesExecutionBackend subclass that creates Jobs its own
way therefore inherits adopts_active_tasks = True and is refused by run_scoring until it either clears
outputs and sets clears_outputs_on_fresh_start = True, or never adopts and sets adopts_active_tasks = False.
A subclass that only overrides build_manifest, or whose submit calls super().submit(spec) and sets the
attribute, keeps the base class's clearing.
Register the entry point¶
Register the class under the altar.execution_backends group, with its name as the key:
A run spec selects a backend with type in its [backend] section. The run
driver passes the section's options to the constructor as keywords, and adds the run's storage when the
constructor takes a storage argument. A backend used from a run spec therefore takes only plain options and
builds its provider client from them. The shipped Kubernetes and Modal backends do this, and also accept an
injected client, which their tests use. LabQueueBackend above takes only an injected client, so your own code
drives it from Python with run_scoring(); see Score a cohort.
Test with the conformance suite¶
Subclass ExecutionBackendContract from altar.testing. Give it a backend fixture wired to a fake provider,
and implement the hooks that script the fake:
import pytest
from altar.testing import ExecutionBackendContract
from lab_queue_backend import LabQueueBackend
class FakeLabQueue:
"""Stand in for the queue's client: jobs wait in memory until a test finishes them."""
def __init__(self) -> None:
self.jobs: dict[str, dict] = {}
async def launch(self, **job) -> str:
job_id = f"job-{len(self.jobs)}"
self.jobs[job_id] = {**job, "state": "queued"}
return job_id
async def describe(self, job_id: str) -> dict | None:
return self.jobs.get(job_id)
class TestLabQueueBackend(ExecutionBackendContract):
@pytest.fixture
def backend(self) -> LabQueueBackend:
return LabQueueBackend(FakeLabQueue())
def finish_task(self, backend, handle, *, exit_code: int) -> None:
backend.client.jobs[handle.external_id].update(state="finished", exit_code=exit_code)
def expire_task(self, backend, handle) -> None:
backend.client.jobs[handle.external_id]["state"] = "timed_out"
def launched_timeout(self, backend, handle) -> int | None:
return backend.client.jobs[handle.external_id]["timeout_s"]
The hooks:
finish_task(backend, handle, exit_code=)makes the task exit with that code.expire_task(backend, handle)makes the task run past its time limit, as the provider would show it.launched_timeout(backend, handle)returns the limit the provider would enforce, including any default it substitutes when none is passed. If the provider cannot run a task without a limit, set the contract'sunlimited_timeout_sto its maximum.fail_input_verification(backend, handle)makes the task fail its own input check. Only a"task"backend implements it.write_task_log(backend, handle, lines)writes a log, for a backend whosefailure_detailreads one.write_task_outputandtask_output, andstart_taskwhen tasks clear their own outputs as they start, are for a backend that declaresadopts_active_tasksorclears_outputs_on_fresh_start. These hooks are provisional.
The suite checks submission, resubmission of the same spec, polling order, success and failure outcomes, failure reasons, missing tasks, timeouts, labels, input digest verification and output clearing. Tests that need a hook you did not implement are skipped. Add your own tests for capabilities the provider cannot represent. Then run one model's plan through your backend and through a shipped backend, and compare the commands and outputs.
Ready when¶
submitpasses the image, command, resources, time limit and labels through unchanged, and never raises for a spec submitted before.pollreportsexit_<n>,timeout,oom_killedandcontainer_missingfailures with the reasons onTaskFailure.TaskHandle.external_idis enough to poll the task after the submitting process restarts.input_digest_verificationis set, and a mismatched input never reaches the task's command.- A GPU request either gets the exact device count or fails.
- The entry point's key equals
name. ExecutionBackendContractpasses against a fake provider.- The page that documents the backend says what it was tested against: fakes, an emulator or a real system.
Next¶
- Validate an implementation lists every conformance suite.
- Package and register a plugin covers entry-point groups and the plugin registry.