Run tasks on a GPU or a cluster¶
A model that runs in containers sends its tasks to the execution backend that the spec's [backend] section
names, or to the one of its [backends.<name>] that the model names. The local backend runs them with Docker on this machine, on its GPU or its CPU. The kubernetes and
modal backends run them on a Kubernetes cluster or on Modal.
The local backend is Stable; kubernetes and modal are Preview. Platform
support gives each backend's test evidence.
AlphaGenome runs in the altar process and needs no backend. altar models <type> says whether a model runs in
a container or inline, and the compute its tasks ask for.
Before you start¶
- A spec that scores with a container model; see Score variants with a model. The examples continue
from its
score.toml. - The model images are built for
linux/amd64. They are 273 MB (LegNet) to 6.1 GB (ChromBPNet) compressed, and each backend pulls them on first use. - For the local backend: Docker, usable by your user. For a GPU model, an NVIDIA GPU and the NVIDIA Container Toolkit.
- For Kubernetes:
pip install "altar[kubernetes]", a kubeconfig or in-cluster credentials, and a volume every pod can mount. - For Modal:
pip install "altar[modal]"and a Modal account whose credentials the Modal client finds, such as the tokenmodal setupwrites.
Run on a local GPU¶
[backend] type = "local" runs each task with docker run on this machine:
data_rootis the directory mounted into every container at/mnt/volume. The run copies each task's inputs there and reads its outputs back. It defaults tobackend-datain the cache,.altar/backend-data.options = { data_root = "..." }names another; a relative path is made absolute against the directoryaltar runstarts in.- A task that asks for a GPU starts with
docker run --gpus, which needs the NVIDIA Container Toolkit. - The run scores one model's batches one after another.
On Linux, each container runs as the owner of data_root, or as you when the directory does not exist yet, so it
can write its outputs there. On macOS, it runs as the image's user, because Docker Desktop maps file ownership
itself. Set container_user to a uid:gid to run as someone else, such as on a root-squashed NFS data_root:
[backend]
type = "local"
options = { data_root = "/nfs/altar-volume", container_user = "1000:1000" }
altar run --validate (Provisional) checks that the run can
create and write data_root. It looks each model's image up with
docker image inspect, then docker manifest inspect. An image it finds in neither place is a warning, since the
registry may be out of reach only for now.
Run a GPU model on the CPU¶
cpu_only = true drops each task's GPU request, so a GPU model runs on the CPU, much more slowly:
Use it for a small smoke run on a machine without a GPU. LegNet asks for no GPU, so it needs no cpu_only.
Give tasks their own GPUs¶
On a machine with several GPUs, pools names sets of devices. Each task gets as many of its pool's free devices as
it asks for, to itself, and a task that finds too few free waits until a running one finishes:
Here every GPU task runs on devices 1, 2 or 3, one task per device, and device 0 stays free for other work. The
shipped models ask for the pool gpu. Without pools, a task gets docker run --gpus <count>, and Docker picks
the devices.
Run on Kubernetes¶
Preview
The Kubernetes backend passes contract tests against a fake cluster. It has not been run from a spec on a live cluster.
type = "kubernetes" runs each task as a Kubernetes Job. Every pod mounts one shared volume at /mnt/volume,
and the run copies inputs to that volume and outputs back from it through the run's [storage]. volume_uri
names where that volume is in the run's storage: a directory this machine and the pods share, or a bucket prefix
that a CSI or gcsfuse volume mounts in the pods.
[backend]
type = "kubernetes"
[backend.options]
namespace = "altar"
pvc_name = "altar-data"
volume_uri = "/shared/altar-volume"
pool_node_selectors = { gpu = { "cloud.google.com/gke-accelerator" = "nvidia-l4" } }
namespaceis where the Jobs are created.pvc_nameis the PersistentVolumeClaim every pod mounts.pool_node_selectorssends a task that asks for a GPU pool to matching nodes. The shipped models ask for the poolgpu. A pool with no entry gets only thenvidia.com/gpucount.run_as_user,run_as_groupandfs_groupset the pod's user and groups, for a volume that needs them.
The backend reads the in-cluster service account when it has one, or else the default kubeconfig or
$KUBECONFIG. A spec without volume_uri stops with
storage and volume_uri select the staged mode together; set both or neither.
kubernetes options lists every option and its default.
A Job is named by a hash of its task. A rerun of the same task adopts a Job that is still running, and replaces one that has finished. Kubernetes deletes a finished Job a day after it ends, which leaves time to read a failed pod's log.
Run on Modal¶
Preview
The Modal backend passes contract tests against a fake Modal client. It has not been run from a spec on a live Modal account.
type = "modal" runs each task as a Modal Sandbox. A Modal Volume holds the shared files, mounted at
/mnt/volume in every sandbox, and the run uploads each task's inputs to it and downloads its outputs.
[backend]
type = "modal"
options = { app_name = "altar-exec", volume_name = "altar-data", pool_accelerators = { gpu = "L40S" } }
app_nameis the Modal App the sandboxes belong to. It is created on first use.volume_nameis the Modal Volume. It is created on first use.pool_acceleratorsmaps a GPU pool to a Modal GPU type.
modal options lists every option and its default.
Modal caps a sandbox at 24 hours, so a task runs for at most that long.
Run different models on different machines or pools¶
[backends.<name>] declares several backends in place of [backend], each with the same keys. Each container
model names the one it runs on with backend, and picks a GPU pool that backend defines with pool. Here Borzoi
uses the A100s of a lab server, nandi, and Enformer its other GPUs:
[backends.nandi]
type = "local"
options = { pools = { gpu = { devices = [1, 2, 3] }, a100 = { devices = [4, 5] } } }
[models.borzoi]
type = "BORZOI"
backend = "nandi"
pool = "a100"
[models.enformer]
type = "ENFORMER"
backend = "nandi"
Enformer keeps the pool its binding asks for, gpu. A second section, such as [backends.cluster] with
type = "kubernetes", sends a model elsewhere; a model on another backend scores after the first one finishes.
pool can also differ per stage, keyed by the binding's operations: pool = { score = "a100", interpret = "gpu" }.
altar prepare and altar interpret run a model's preparation and interpretation on its backend too.
Each backend defines its own pools: local its pools option, kubernetes gpu and the names in
pool_node_selectors, and modal the keys of pool_accelerators
(GPU pools). A pool the backend does not define stops the run before anything
is submitted:
altar run: error: [models.borzoi] pool: [backends.nandi] (local) defines no GPU pool 'h100'; it defines: a100, gpu
Where a model runs is not part of its identity: scores stored from one backend are reused on another. The run
summary and the export's provenance record each model's backend and pools under compute.
Check the result¶
altar run --validate score.toml builds the backend from the spec's options without starting a task, and names
each model's backend and GPU pools. An option the backend does not take stops it with exit status 2:
altar run: error: backend: 'modal' cannot be built from the spec's options (ModalExecutionBackend.__init__() got an unexpected keyword argument 'app_nme')
With the local backend, --validate also checks data_root and looks up each image.
After a run, each model's line on standard error says how many variants it scored in how many batches, as in Run it.
If it fails¶
- A container model in a spec with no
[backend]stops before it reads the cohort, and the message says to add[backend]withtype = "local". - A task that fails ends the run with exit status 1. The backend reports a reason, such as
exit_1,timeout,oom_killedorinput_digest_mismatch; run again with-vto see where it stopped. - Tasks still open after
[backend] timeout, 24 hours unless the spec sets it, end the run with exit status 1. The tasks ran too long says what happens to them on each backend.
Fix a run that stops or warns covers the other messages. A rerun of the same spec sends only the variants the store does not hold yet.
Next¶
- Choose where results are stored keeps the scores, and copies files through a bucket.
- Prepare model resources runs a model's preparation step on the same backend.
- Run container tasks builds a backend in Python, for a program that drives execution itself.
- Add an execution backend writes a backend of your own.
- Tutorial 1: score variants locally with Cherimoya runs a model in local Docker, on a GPU or the CPU, in a notebook.