Skip to content

Run tasks on a GPU or a cluster

A model that runs in containers sends its tasks to the execution backend that the spec's [backend] section names, or to the one of its [backends.<name>] that the model names. The local backend runs them with Docker on this machine, on its GPU or its CPU. The kubernetes and modal backends run them on a Kubernetes cluster or on Modal.

The local backend is Stable; kubernetes and modal are Preview. Platform support gives each backend's test evidence.

AlphaGenome runs in the altar process and needs no backend. altar models <type> says whether a model runs in a container or inline, and the compute its tasks ask for.

Before you start

  • A spec that scores with a container model; see Score variants with a model. The examples continue from its score.toml.
  • The model images are built for linux/amd64. They are 273 MB (LegNet) to 6.1 GB (ChromBPNet) compressed, and each backend pulls them on first use.
  • For the local backend: Docker, usable by your user. For a GPU model, an NVIDIA GPU and the NVIDIA Container Toolkit.
  • For Kubernetes: pip install "altar[kubernetes]", a kubeconfig or in-cluster credentials, and a volume every pod can mount.
  • For Modal: pip install "altar[modal]" and a Modal account whose credentials the Modal client finds, such as the token modal setup writes.

Run on a local GPU

[backend] type = "local" runs each task with docker run on this machine:

[backend]
type = "local"
  • data_root is the directory mounted into every container at /mnt/volume. The run copies each task's inputs there and reads its outputs back. It defaults to backend-data in the cache, .altar/backend-data. options = { data_root = "..." } names another; a relative path is made absolute against the directory altar run starts in.
  • A task that asks for a GPU starts with docker run --gpus, which needs the NVIDIA Container Toolkit.
  • The run scores one model's batches one after another.

On Linux, each container runs as the owner of data_root, or as you when the directory does not exist yet, so it can write its outputs there. On macOS, it runs as the image's user, because Docker Desktop maps file ownership itself. Set container_user to a uid:gid to run as someone else, such as on a root-squashed NFS data_root:

[backend]
type = "local"
options = { data_root = "/nfs/altar-volume", container_user = "1000:1000" }

altar run --validate (Provisional) checks that the run can create and write data_root. It looks each model's image up with docker image inspect, then docker manifest inspect. An image it finds in neither place is a warning, since the registry may be out of reach only for now.

Run a GPU model on the CPU

cpu_only = true drops each task's GPU request, so a GPU model runs on the CPU, much more slowly:

[backend]
type = "local"
cpu_only = true

Use it for a small smoke run on a machine without a GPU. LegNet asks for no GPU, so it needs no cpu_only.

Give tasks their own GPUs

On a machine with several GPUs, pools names sets of devices. Each task gets as many of its pool's free devices as it asks for, to itself, and a task that finds too few free waits until a running one finishes:

[backend]
type = "local"
options = { pools = { gpu = { devices = [1, 2, 3] } } }

Here every GPU task runs on devices 1, 2 or 3, one task per device, and device 0 stays free for other work. The shipped models ask for the pool gpu. Without pools, a task gets docker run --gpus <count>, and Docker picks the devices.

Run on Kubernetes

Preview

The Kubernetes backend passes contract tests against a fake cluster. It has not been run from a spec on a live cluster.

type = "kubernetes" runs each task as a Kubernetes Job. Every pod mounts one shared volume at /mnt/volume, and the run copies inputs to that volume and outputs back from it through the run's [storage]. volume_uri names where that volume is in the run's storage: a directory this machine and the pods share, or a bucket prefix that a CSI or gcsfuse volume mounts in the pods.

[backend]
type = "kubernetes"

[backend.options]
namespace = "altar"
pvc_name = "altar-data"
volume_uri = "/shared/altar-volume"
pool_node_selectors = { gpu = { "cloud.google.com/gke-accelerator" = "nvidia-l4" } }
  • namespace is where the Jobs are created.
  • pvc_name is the PersistentVolumeClaim every pod mounts.
  • pool_node_selectors sends a task that asks for a GPU pool to matching nodes. The shipped models ask for the pool gpu. A pool with no entry gets only the nvidia.com/gpu count.
  • run_as_user, run_as_group and fs_group set the pod's user and groups, for a volume that needs them.

The backend reads the in-cluster service account when it has one, or else the default kubeconfig or $KUBECONFIG. A spec without volume_uri stops with storage and volume_uri select the staged mode together; set both or neither. kubernetes options lists every option and its default.

A Job is named by a hash of its task. A rerun of the same task adopts a Job that is still running, and replaces one that has finished. Kubernetes deletes a finished Job a day after it ends, which leaves time to read a failed pod's log.

Run on Modal

Preview

The Modal backend passes contract tests against a fake Modal client. It has not been run from a spec on a live Modal account.

type = "modal" runs each task as a Modal Sandbox. A Modal Volume holds the shared files, mounted at /mnt/volume in every sandbox, and the run uploads each task's inputs to it and downloads its outputs.

[backend]
type = "modal"
options = { app_name = "altar-exec", volume_name = "altar-data", pool_accelerators = { gpu = "L40S" } }
  • app_name is the Modal App the sandboxes belong to. It is created on first use.
  • volume_name is the Modal Volume. It is created on first use.
  • pool_accelerators maps a GPU pool to a Modal GPU type.

modal options lists every option and its default.

Modal caps a sandbox at 24 hours, so a task runs for at most that long.

Run different models on different machines or pools

[backends.<name>] declares several backends in place of [backend], each with the same keys. Each container model names the one it runs on with backend, and picks a GPU pool that backend defines with pool. Here Borzoi uses the A100s of a lab server, nandi, and Enformer its other GPUs:

[backends.nandi]
type = "local"
options = { pools = { gpu = { devices = [1, 2, 3] }, a100 = { devices = [4, 5] } } }

[models.borzoi]
type = "BORZOI"
backend = "nandi"
pool = "a100"

[models.enformer]
type = "ENFORMER"
backend = "nandi"

Enformer keeps the pool its binding asks for, gpu. A second section, such as [backends.cluster] with type = "kubernetes", sends a model elsewhere; a model on another backend scores after the first one finishes. pool can also differ per stage, keyed by the binding's operations: pool = { score = "a100", interpret = "gpu" }. altar prepare and altar interpret run a model's preparation and interpretation on its backend too.

Each backend defines its own pools: local its pools option, kubernetes gpu and the names in pool_node_selectors, and modal the keys of pool_accelerators (GPU pools). A pool the backend does not define stops the run before anything is submitted:

altar run: error: [models.borzoi] pool: [backends.nandi] (local) defines no GPU pool 'h100'; it defines: a100, gpu

Where a model runs is not part of its identity: scores stored from one backend are reused on another. The run summary and the export's provenance record each model's backend and pools under compute.

Check the result

altar run --validate score.toml builds the backend from the spec's options without starting a task, and names each model's backend and GPU pools. An option the backend does not take stops it with exit status 2:

altar run: error: backend: 'modal' cannot be built from the spec's options (ModalExecutionBackend.__init__() got an unexpected keyword argument 'app_nme')

With the local backend, --validate also checks data_root and looks up each image.

After a run, each model's line on standard error says how many variants it scored in how many batches, as in Run it.

If it fails

  • A container model in a spec with no [backend] stops before it reads the cohort, and the message says to add [backend] with type = "local".
  • A task that fails ends the run with exit status 1. The backend reports a reason, such as exit_1, timeout, oom_killed or input_digest_mismatch; run again with -v to see where it stopped.
  • Tasks still open after [backend] timeout, 24 hours unless the spec sets it, end the run with exit status 1. The tasks ran too long says what happens to them on each backend.

Fix a run that stops or warns covers the other messages. A rerun of the same spec sends only the variants the store does not hold yet.

Next