Skip to content

Add an execution backend

An execution backend runs container tasks on a compute system: it submits each task and reports its state. A task is one unit of work, such as one model fold in one container. The backend runs the command a model binding built, unchanged, and never reads the model's scores.

When to choose an execution backend

Write one when your tasks must run somewhere the shipped backends don't reach, such as a lab's batch queue or another cloud's container service. Altar ships local (Docker on one machine), kubernetes and modal; Platform support says how each is tested. Every model binding's plan runs on every backend, so a new backend needs no change to any binding. Choose an interface lists the other extension points.

The interface

Subclass ExecutionBackend, imported from altar.execution. It is Stable.

Member Required? What it is
name required The registry name, and the type of a run spec's [backend] section.
submit(spec) required Launches one ContainerTaskSpec and returns a TaskHandle that still works after the submitting process restarts.
poll(handles) required Returns one current TaskStatus per handle, in order, without blocking.
input_digest_verification required: "staging" or "task" Where the backend checks a content-addressed input. The default, None, fails the conformance suite.
stage_inputs(transfers), collect_outputs(transfers) default: do nothing Copy inputs to where the task reads them, and outputs back to their URIs.
path_resolver() default: paths unchanged Turns a logical path into the path the task sees, such as /mnt/volume/jobs/....
failure_detail(handle) default None The last lines of a failed task's log.
cancel(handle) default: do nothing Stops or deletes a task. A missing task counts as cancelled.
capacity() default: an empty report Free slots per pool, for a scheduler of your own. Altar never reads it.
gpu_pools() provisional; default None, any pool The GPU pool names the backend serves. altar run refuses a model whose pool is not one of them before anything is submitted; see Declare your GPU pools.
await_completion(handles), recover(failures, banned=) default: not supported, do nothing For a backend with a native wait, or one that can restart failed tasks.
adopts_active_tasks, clears_outputs_on_fresh_start, writes_output_in_place(transfer) provisional; default False Who deletes outputs an earlier run left; see Clear outputs an earlier run left.

A provisional member is expected to change; see Provisional names.

A minimal backend

This backend runs each task as a job on a lab's batch queue, through the queue's own client. The queue's workers mount the lab's shared filesystem, so a task reads each input at its path, and stage_inputs checks the bytes there. Put it in a module of your own, such as lab_queue_backend.py:

from typing import Any

from altar.execution import (
    ContainerTaskSpec,
    ExecutionBackend,
    TaskFailure,
    TaskHandle,
    TaskState,
    TaskStatus,
    Transfer,
    verify_transfer_digest,
)


class LabQueueBackend(ExecutionBackend):
    """Run each task as one job on the lab's batch queue, whose workers mount the lab's shared filesystem."""

    name = "lab_queue"
    input_digest_verification = "staging"

    def __init__(self, client: Any) -> None:
        self.client = client  # the queue's client; the caller created it, so the caller closes it

    async def stage_inputs(self, transfers: list[Transfer]) -> None:
        # Workers read each input at its path on the shared filesystem, so check the bytes there.
        for transfer in transfers:
            verify_transfer_digest(transfer, transfer.uri)

    async def submit(self, spec: ContainerTaskSpec) -> TaskHandle:
        job_id = await self.client.launch(
            image=spec.image,
            command=spec.command,
            gpu_type=spec.resources.gpu,
            gpu_count=spec.resources.count if spec.resources.gpu else 0,
            memory_gb=spec.resources.memory_gb,
            timeout_s=spec.timeout_s,
            labels=spec.labels,
        )
        return TaskHandle(
            backend=self.name, external_id=job_id, labels=dict(spec.labels), plugin_identity=spec.plugin_identity
        )

    async def poll(self, handles: list[TaskHandle]) -> list[TaskStatus]:
        return [self._status(handle, await self.client.describe(handle.external_id)) for handle in handles]

    def _status(self, handle: TaskHandle, job: dict[str, Any] | None) -> TaskStatus:
        if job is None:
            failure = TaskFailure("container_missing", "the queue has no such job")
            return TaskStatus(handle, TaskState.FAILED, failure)
        if job["state"] in ("queued", "running"):
            return TaskStatus(handle, TaskState.PENDING if job["state"] == "queued" else TaskState.RUNNING)
        if job["state"] == "timed_out":
            return TaskStatus(handle, TaskState.FAILED, TaskFailure("timeout", "the job ran past its time limit"))
        code = job["exit_code"]
        if code == 0:
            return TaskStatus(handle, TaskState.SUCCEEDED)
        failure = TaskFailure(f"exit_{code}", f"the job exited with code {code}", deterministic=True, exit_code=code)
        return TaskStatus(handle, TaskState.FAILED, failure)

What each part does:

  • submit passes the image, the command, the resource request, the time limit and the labels to the queue unchanged, and returns a handle whose external_id is the queue's job ID.
  • poll asks the queue about each handle and maps its state to an Altar TaskState. A job the queue no longer knows is a terminal failure with reason container_missing.
  • stage_inputs checks each content-addressed input with verify_transfer_digest, which raises TransferIntegrityError on a mismatch, a missing file or a directory.

A queue whose workers can't see the submitting machine's files copies each input to storage the workers read in stage_inputs, hashing the bytes it writes, and copies each output back in collect_outputs.

Honor the task declaration

The backend must map:

  • image and command without model-specific rewrites;
  • CPU, memory, and GPU requests, with ResourceRequest.count as the exact number of devices;
  • logical input and output transfers;
  • structured labels;
  • timeout_s as a wall-clock limit enforced by the provider or by poll, and None as no limit. If the provider caps task lifetime or applies a default when no limit is passed, request its maximum for None and reject a longer explicit limit;
  • provider state into Altar task states.

Keep task labels unchanged: the code that drives a run groups tasks and decides readiness by them. If the provider cannot represent something the spec asks for, fail the submission. A task that asks for a GPU must never run on a CPU instead.

Verify content-addressed inputs

An input Transfer with a digest names exact bytes, such as model weights: the SHA-256 of one regular file. A digest never covers a directory, so a digest-bearing input that resolves to a directory or a missing file is rejected like a mismatch. Every backend verifies digests before the task's command runs, and declares where with the class attribute input_digest_verification:

  • "staging": stage_inputs hashes the bytes it copies and raises TransferIntegrityError on a mismatch, as the local and Modal backends do.
  • "task": each task checks its own inputs first and fails with reason input_digest_mismatch. Use this when inputs already sit on a shared volume the submitting process cannot read; the Kubernetes backend adds an init container that runs verify_files_script() on the mounted paths.

The conformance suite fails a backend that declares neither.

Hash once per volume

A large input such as a reference genome is read by every task of every job. Hashing it in full before each task can take longer than the task itself, so a verified file carries a verification record: a one-line file beside it, at its path plus .sha256-verified.

verified-file/1 sha256:<64 hex> size=<bytes> mtime=<seconds> ctime=<seconds> inode=<number>

The line names the digest the bytes matched, and the file's size, whole-second modification and status-change times, and inode number when they matched. Rewriting, replacing, or truncating the file changes at least one of those values, so the record no longer matches and the next check hashes the file again. A record is only written once the file's status-change second has passed, and only if the file did not change while it was hashed.

  • verify_file_digest(path, digest) accepts a file from a current record, or hashes it and writes one.
  • verify_transfer_digest(transfer, path, record=True) always hashes, then writes a record. Use it where bytes enter storage your tasks read, and only for copies the backend owns.
  • verify_files_script() is the same check as a POSIX shell program, for BusyBox or GNU coreutils.

The local backend records each verified copy and skips copying an input whose destination already holds a recorded copy of the same digest. The Kubernetes init container accepts recorded inputs and records the ones it hashes. Model runtimes check their inputs the same way, so once a resource is staged and recorded, each fold task reads only its metadata.

Anyone who can write the volume can also write a record. Where untrusted parties can write it, construct the Kubernetes backend with trust_verification_records=False, which hashes every input in every task. Its init container still writes a record after each successful hash, so a runtime that trusts records then accepts the file from the record its own task's verifier just wrote instead of hashing it a second time.

Report task state

Use the reason codes documented on TaskFailure for the same events on every backend: exit_<n> with exit_code for an application exit, timeout, oom_killed, input_digest_mismatch, container_missing for a task the provider can no longer find, and job_failed when the provider gives no finer detail. A missing task must be a terminal failure, never a task that stays running forever.

TaskHandle.external_id must be enough to poll the provider after the submitting process restarts. Use the provider's own task ID, not a display name, and keep grouping information in the labels.

Accept resubmission

A failed or interrupted run is retried by submitting the same specs again, so submit must never raise because a spec was submitted before. A backend whose provider mints a fresh ID per task, as Docker and Modal do, launches another task. A backend that names tasks by their spec must settle the name conflict itself:

  • a task that is still pending or running is adopted: submit returns a handle to it, so the same work never runs twice at once;
  • a task that has finished, whether it succeeded or failed, is deleted and submitted again. A spec names its inputs by path, and a new run under the same job can put different content at the same path (a new variant batch, for example), so a finished task's output is never taken as the new run's.

How the Kubernetes backend resubmits

The Kubernetes backend names each Job by a hash of its spec and follows these rules:

  • Its runtime reports a taken name as K8sJobExistsError (the API server's 409 AlreadyExists) and reads the existing Job with read_job. A custom KubernetesRuntime must raise K8sJobExistsError for a name conflict and report each Job's annotations, and should report its uid.
  • Each Job records a hash of its fully rendered manifest in the altar.kundajelab.org/manifest-sha256 annotation. The name covers only the spec, so an active Job whose manifest differs, for example one created by a backend with another PVC, verifier image, or node selector, is neither adopted nor deleted: submit raises K8sJobExistsError. Let it finish, or delete it, before resubmitting.
  • An active Job without that annotation is refused the same way. A custom KubernetesRuntime must return annotations from list_jobs and read_job.
  • A finished Job is deleted with its pods before it is created again. The delete is conditioned on the Job's uid, so when two drivers resubmit at once, neither deletes the fresh Job the other just created; the slower one adopts it instead.
  • If the existing Job disappears between the conflict and the read, or is still being deleted, the create is retried with backoff for about a minute before submit raises K8sJobExistsError.

An adopted Job runs with whatever its input paths hold. Content-addressed inputs (a Transfer.digest) are verified before it starts, but inputs named only by path are not. So if a driver crashes and a new run writes different content to the same path before the old Job reads it, the adopted Job reads the new content.

Declare your GPU pools

ResourceRequest.gpu names a pool, such as the shipped models' gpu, and a spec's [models.<name>] pool picks another one. A backend whose pools come from its own options returns their names from gpu_pools(). altar run, altar prepare and altar interpret then check every model's pools against them and stop, naming the pools the backend defines, before any task is submitted. Call require_gpu_pool(self, spec.resources.gpu) at the top of submit so a program that submits directly gets the same UnknownPoolError:

from altar.execution import require_gpu_pool


class LabQueueBackend(ExecutionBackend):
    ...

    def gpu_pools(self) -> frozenset[str]:
        return frozenset(self.client.gpu_types())  # the queue's own names, such as "gpu" and "a100"

    async def submit(self, spec: ContainerTaskSpec) -> TaskHandle:
        require_gpu_pool(self, spec.resources.gpu)
        ...

The default, None, accepts every name, so a backend that does not override it keeps its own handling of an unknown pool. The conformance suite checks that a backend declaring its pools refuses any other at submit.

Clear outputs an earlier run left

A task must never hand back a file an earlier run left at one of its outputs. Two class attributes on ExecutionBackend say who deletes those files:

  • adopts_active_tasks: submit may return a handle to a task that is still pending or running instead of launching a new one. The default is False.
  • clears_outputs_on_fresh_start: submit deletes the file at each of spec.outputs before it launches a fresh attempt, and keeps the outputs of a task it adopts. The default is False.

Both are provisional. A backend that adopts also says, through the provisional method writes_output_in_place(transfer), whether its task writes an output's uri directly. The default answer is False: collect_outputs copies the output there after the task succeeds, so a driver can delete that location before submitting without losing an adopted task's work.

run_scoring reads them before it submits anything:

adopts_active_tasks clears_outputs_on_fresh_start What run_scoring does
False either Deletes each terminal output's file through its Storage, then submits.
True True Deletes each terminal output that writes_output_in_place rejects, and leaves the rest to the backend. A restarted driver collects what an adopted task wrote.
True False Raises ValueError before anything is submitted.

A backend whose provider launches a new task on every submit, as Docker and Modal do, needs neither attribute; the reference backends still declare clears_outputs_on_fresh_start, because their submit deletes the outputs on the backend's own volume. A backend that adopts declares both, and clears only when it commits to a new attempt. The Kubernetes backend does it inside the new Job: a clear-outputs init container deletes the files at the outputs' mounted paths before the task starts, so an adopted Job, which starts no new pod, keeps what it wrote. In the staged mode it also deletes them through its Storage before it creates the Job. A failure of that init container is reported with reason output_clear_failed.

The Kubernetes backend writes an output in place when its uri is the output's volume location (staged mode), or the path the pod writes, <mount_root>/<logical_path> (pre-staged mode). In the pre-staged mode nothing copies an output anywhere else, so run_scoring deletes a file an earlier run left at any other location and the run fails as not produced. A driver that sees the volume at a path other than mount_root should use the staged mode, with volume_uri set to that path: outputs already at their volume location are not copied.

The clearing lives in submit, so a subclass that overrides submit reads clears_outputs_on_fresh_start as False until its own class body sets it again. A KubernetesExecutionBackend subclass that creates Jobs its own way therefore inherits adopts_active_tasks = True and is refused by run_scoring until it either clears outputs and sets clears_outputs_on_fresh_start = True, or never adopts and sets adopts_active_tasks = False. A subclass that only overrides build_manifest, or whose submit calls super().submit(spec) and sets the attribute, keeps the base class's clearing.

Register the entry point

Register the class under the altar.execution_backends group, with its name as the key:

[project.entry-points."altar.execution_backends"]
lab_queue = "lab_queue_backend:LabQueueBackend"

A run spec selects a backend with type in its [backend] section. The run driver passes the section's options to the constructor as keywords, and adds the run's storage when the constructor takes a storage argument. A backend used from a run spec therefore takes only plain options and builds its provider client from them. The shipped Kubernetes and Modal backends do this, and also accept an injected client, which their tests use. LabQueueBackend above takes only an injected client, so your own code drives it from Python with run_scoring(); see Score a cohort.

Test with the conformance suite

Subclass ExecutionBackendContract from altar.testing. Give it a backend fixture wired to a fake provider, and implement the hooks that script the fake:

import pytest
from altar.testing import ExecutionBackendContract

from lab_queue_backend import LabQueueBackend


class FakeLabQueue:
    """Stand in for the queue's client: jobs wait in memory until a test finishes them."""

    def __init__(self) -> None:
        self.jobs: dict[str, dict] = {}

    async def launch(self, **job) -> str:
        job_id = f"job-{len(self.jobs)}"
        self.jobs[job_id] = {**job, "state": "queued"}
        return job_id

    async def describe(self, job_id: str) -> dict | None:
        return self.jobs.get(job_id)


class TestLabQueueBackend(ExecutionBackendContract):
    @pytest.fixture
    def backend(self) -> LabQueueBackend:
        return LabQueueBackend(FakeLabQueue())

    def finish_task(self, backend, handle, *, exit_code: int) -> None:
        backend.client.jobs[handle.external_id].update(state="finished", exit_code=exit_code)

    def expire_task(self, backend, handle) -> None:
        backend.client.jobs[handle.external_id]["state"] = "timed_out"

    def launched_timeout(self, backend, handle) -> int | None:
        return backend.client.jobs[handle.external_id]["timeout_s"]

The hooks:

  • finish_task(backend, handle, exit_code=) makes the task exit with that code.
  • expire_task(backend, handle) makes the task run past its time limit, as the provider would show it.
  • launched_timeout(backend, handle) returns the limit the provider would enforce, including any default it substitutes when none is passed. If the provider cannot run a task without a limit, set the contract's unlimited_timeout_s to its maximum.
  • fail_input_verification(backend, handle) makes the task fail its own input check. Only a "task" backend implements it.
  • write_task_log(backend, handle, lines) writes a log, for a backend whose failure_detail reads one.
  • write_task_output and task_output, and start_task when tasks clear their own outputs as they start, are for a backend that declares adopts_active_tasks or clears_outputs_on_fresh_start. These hooks are provisional.

The suite checks submission, resubmission of the same spec, polling order, success and failure outcomes, failure reasons, missing tasks, timeouts, labels, input digest verification and output clearing. Tests that need a hook you did not implement are skipped. Add your own tests for capabilities the provider cannot represent. Then run one model's plan through your backend and through a shipped backend, and compare the commands and outputs.

Ready when

  • submit passes the image, command, resources, time limit and labels through unchanged, and never raises for a spec submitted before.
  • poll reports exit_<n>, timeout, oom_killed and container_missing failures with the reasons on TaskFailure.
  • TaskHandle.external_id is enough to poll the task after the submitting process restarts.
  • input_digest_verification is set, and a mismatched input never reaches the task's command.
  • A GPU request either gets the exact device count or fails.
  • The entry point's key equals name.
  • ExecutionBackendContract passes against a fake provider.
  • The page that documents the backend says what it was tested against: fakes, an emulator or a real system.

Next