Skip to content

Fix a run that stops or warns

These are the messages altar run prints when it stops or warns, and what to do about each. Most also apply to altar prepare and altar interpret, which print the same messages under their own names.

Read what the run reports

altar run prints its progress on standard error, one line per stage, and the run summary as JSON on standard output. A progress line starts with the spec section it belongs to, such as [variants]. A problem starts with altar run: warning: when the run went on past it, or altar run: error: when it stopped the run. An error says what went wrong and, for a problem in the spec, names the section to change:

altar run: error: [sources.gnomad] type 'gnomd' is not an installed annotation source; did you mean 'gnomad'? Installed: alphamissense, bigquery, clinvar, gnomad, gpnstar, regions, spliceai, sqlite, variant_gene_links.

A spec that does not parse lists one problem per indented line, each with the path of the key:

altar run: error: invalid run spec:
  run.job_id: String should match pattern '^[A-Za-z0-9][A-Za-z0-9._-]*$'
  variants.formt: Extra inputs are not permitted
  exports: Extra inputs are not permitted

The exit status says what kind of stop it was:

Status Meaning Next step
0 The run finished. Warnings may still have been printed. Read The run warns.
2 The run refused the spec, a file, or a missing extra or credential. Change what the message names; see The run refuses the spec.
1 The run failed partway. Rerun with -v; see The run fails.
130 You interrupted it with Ctrl-C. The export was not written. Run the same command again.

Exit statuses lists the statuses of every command.

Three flags help:

  • altar run --validate spec.toml (Provisional) makes every check that needs no variant, then stops. It reads no variant and writes no file, so it is the fastest way to find a spec problem.
  • -v (--verbose) prints debug lines, the full traceback when the run stops, and the time on every line. A failure the run did not foresee ends with Run again with -v to see where it stopped.
  • -q (--quiet) prints only warnings and errors.

The run refuses the spec

A refusal exits with status 2. Most refusals happen before the cohort is read.

A name, key or type is misspelled

An unknown section or key is refused with its path, as above. A misspelled type gets the installed name it is closest to. A type is matched exactly, so type = "legnet" is refused with did you mean 'LEGNET'?. A model's configuration key gets the same help:

altar run: error: models.legnet-wtc11.configuration: unknown option 'average_revers'; did you mean 'average_reverse'? Valid options: weights, training_config, cell_type, assay, average_reverse, image.

altar models <type> and altar sources <type> list every key a model or a source takes. The run spec reference lists every section and key.

A section is missing

altar run: error: the run spec declares no [models.<name>] and no [sources.<name>], so it has nothing to score or annotate; add a model to score the cohort, or a source to annotate it
altar run: error: [models.legnet-wtc11] (LEGNET) runs in containers, so the spec needs a [backend] to run them; add [backend] with type = "local" to run them with Docker on this machine
altar run: error: models.legnet-wtc11.run_inputs: genome is required; set [variants] reference_fasta, since this model reads the reference genome
altar run: error: models.sei.output selects artifact outputs (chromatin_profile_effects), but the spec declares no [artifact_store] to store them; add one (for example type = "storage")

Add the section the message names. A model whose prioritization reads an annotation column, such as ChromBPNet's region_type, also needs the source that supplies it, and the message names the [sources.<name>] section to add.

A file is not where the spec says

altar run: error: variants.path: cohort.tsv is not a file

A relative path in a spec resolves against the directory you start altar in, not the spec's directory. Start the command from the directory the spec's paths are relative to, or write absolute paths. The same rule places the run's directory, the cache (.altar unless [run] cache_dir or ALTAR_CACHE_DIR names another), a [store] file and the local backend's data_root: a run started from another directory writes them in a new place and reuses no stored score.

A VCF whose name does not end in .vcf, .vcf.gz or .vcf.bgz would be read as a TSV, and is refused:

altar run: error: variants.path: cohort.txt is a VCF, but its name does not end in .vcf, .vcf.gz or .vcf.bgz, so it would be read as a canonical TSV; set [variants] format = "vcf"

A malformed row stops the run with the row it could not read.

The run directory already exists

altar run: error: run.work_dir: the run directory defaults to the job_id, and hg38 already exists and holds files another run did not write. Set [run] work_dir to a new directory, or choose another job_id

Without [run] work_dir, a run writes to a directory named after its job_id. When a directory of that name already holds other files, the run stops before it writes anything. Set work_dir, or rename the job_id. A directory an earlier run with the same job_id wrote is used again.

The genome builds do not match

The run checks every model and source against [run] genome_build before it opens any data, and names every one that cannot serve it:

altar run: error: [sources.gnomad] serves hg38 only; the run is hg19.
[sources.clinvar] reads its publisher's copy, which is hg38; the run is hg19. Point it at hg19 data with its dataset option (its binding documents how to build one).
Remove [sources.gnomad] or lift the cohort over to hg38; if the cohort is already on hg38, set [run] genome_build = "hg38".

Fix genome_build if it is wrong, remove the sources that serve only another build, or point a source at data on the run's build. Give a run its variants explains the check.

The reference digest does not match

altar run: error: variants.reference_digest: ... Restore the expected FASTA, or update reference_digest only if this reference change is intended; reference-reading models will use a different score identity

The FASTA at reference_fasta is not the file the digest names. Restore it, or update the digest if you meant to change the reference. Models that read the reference then score the cohort again.

A credential or an extra is missing

altar run: error: [models.alphagenome] configuration.credential: the environment variable ALPHAGENOME_API_KEY is not set; export it before running
altar run: error: [sources.phylop]: reading hf:// needs the hf extra: pip install "altar[hf]"

Export the variable the spec names, or install the extra. A Google Cloud error about credentials, permissions or a missing project or dataset is a refusal too. Its message ends with what to do, such as running gcloud auth application-default login.

The local backend's data_root is not usable

altar run: error: backend.options.data_root: /data/altar is not writable by this user

Name a directory you can create and write. Without a data_root, the backend uses backend-data in the cache, and the message names run.cache_dir. altar prepare also refuses a data_root inside <cache_dir>/staging, or one that contains it.

Prepared files are missing or stale

A model that reads files a preparation step writes, such as ChromBPNet, needs altar prepare first:

altar run: error: [models.k562] reads files a preparation step writes, and .altar/staging does not have them: ... Run `altar prepare SPEC --model k562` with this spec first; https://kundajelab.github.io/altar/integrations/chrombpnet/ describes it
altar run: error: [models.k562] preparation publication is incomplete; run `altar prepare SPEC --model k562` successfully before scoring, even when scores are already cached
altar run: error: [models.k562] the files `altar prepare` published were prepared with ... Run `altar prepare SPEC --model k562` with this spec, then score again

The first means preparation has not run with this cache: start both commands from the same directory, or give them the same cache_dir. The second means a preparation stopped while it published its files. The third means the model's configuration changed after preparation. In each case run altar prepare with the same spec and the same [models.<name>] key, then score again; see Prepare model resources.

A changed setting meets a store without lineages

altar run: error: [models.legnet-wtc11]: stored scores for model 'legnet-wtc11' use incompatible scientific scoring semantics. The store at legnet-run/scores.sqlite keeps no lineages, so it holds one setting per model key. Give the changed model a new key, such as [models.legnet-wtc11-2], or keep scores in a new store with lineages = true, such as [store] options = { path = "scores-lineages.sqlite", lineages = true }

A store created without lineages holds one set of settings per model key, so it cannot keep a changed setting's scores beside the old ones. Do one of the two things the message names. A lineages option that disagrees with an existing store file is refused too; remove the option to open the file as it was created.

A store cannot be built

altar run: error: store: 'bigquery' cannot be built from the spec's options (BigQueryScoreStore.__init__() missing 1 required positional argument: 'client'); it reaches its backend through a connection, so declare one and name it with store.connection

A bigquery store or detail store needs a gcp connection, named with connection. A connection on a sqlite store or on local storage is refused. Choose where results are stored covers these and a SQLite file in a missing directory.

The run fails

A failure exits with status 1. It is something the checks could not foresee: a container task that failed, a lost connection, a task that ran too long. Rerun with -v to see where it stopped. Scores of the batches that finished before the failure are stored, and the next run reuses them.

A container task fails

The message names the failed task, its exit code, a reason such as exit_1 or oom_killed, and the last lines of the container's log:

altar run: error: ReconcileStalled: reconciliation stalled — group(s) blocked by a failed shard with no on_failed: ...

Read the end of the log in the message. The common causes on the local backend:

Cause Sign Fix
No usable GPU RuntimeError: docker run failed (exit 125): docker run --detach ... --gpus 1 ..., followed by Docker's own message Install the NVIDIA Container Toolkit, or set [backend] cpu_only = true to run the task on the CPU.
The image cannot be pulled RuntimeError: docker run failed ... with Docker's pull error, after a warning that the image was not found Check the network and the image name and digest. The images are built for linux/amd64; see Platform support.
The container cannot write its outputs A task fails, and the log reports a permission error under /mnt/volume On Linux the container runs as the owner of data_root. For a root-squashed NFS data_root, set [backend.options] container_user = "uid:gid" to a user that can write it.
Out of memory The reason is oom_killed Free memory on the machine. On macOS, raise Docker Desktop's memory limit.
The tasks ran too long altar run: error: [models.<name>] tasks still running after 24h (the [backend] timeout) See The tasks ran too long.

A kubernetes backend declared in a spec needs volume_uri in its options. Without it the run stops with ValueError: storage and volume_uri select the staged mode together; set both or neither; see backend.

The tasks ran too long

The run waits for each batch of a model, and for each wave of stages of altar prepare and altar interpret, at most [backend] timeout, 24 hours unless the spec sets it. Tasks still open after that stop the run:

altar run: error: [models.chrombpnet] tasks still running after 24h (the [backend] timeout); 3 tasks remain open, and they were cancelled. Raise [backend] timeout, or lower [models.chrombpnet] max_variants_per_batch so each batch finishes sooner; running the spec again keeps the batches already stored and starts this one over

Raise timeout, for example to "2d", or score fewer variants per batch with the model's max_variants_per_batch. What happens to the tasks still open depends on the backend:

Backend The open tasks Running the spec again
local Stopped and removed (docker rm --force). Starts the batch over in new containers.
kubernetes Their Jobs are deleted, and Kubernetes then removes their pods. Creates the Jobs again, so the batch starts over.
modal Keep running until the task's own time limit, at most Modal's 24 hours, because the backend cannot stop a sandbox. Stop them from the Modal dashboard if you do not want them to finish. Starts new sandboxes; it does not use the ones still running.

In every case the batches that finished before the limit are stored, and a rerun scores only the variants still missing. altar prepare starts the model's preparation over. altar interpret keeps the products of every method that finished and runs the rest. Only a run that is killed outright, so that it cannot cancel anything, leaves Kubernetes Jobs running; a rerun with the same job_id then picks up those still pending or running instead of starting them again.

A spec written for an earlier release may still set [backend] max_ticks, a count of checks. It is refused; set timeout instead.

A source cannot be reached

A source that cannot open its data stops the run before the cohort is read, naming the section:

altar run: error: [sources.gnomad]: could not open the 'gnomad' source: ...

Check the network, and the credentials of the connection the section names. A run that reads a published copy from Hugging Face (location = "hf") can slow down or wait on its rate limit; see Hugging Face rate limits.

An interpretation leaves variants without products

altar interpret exits with status 1 when an operation leaves variants without products:

altar interpret: error: contributions.deepshap_counts left variants without products; see the summary's failure and skipped entries, and run the same command again to retry them

The methods before the failed one stay stored. Run the same command again to retry the missing variants.

The run warns

A warning starts with altar run: warning:. It does not stop the run, and the exit status is 0. -q keeps warnings.

Variants a lookup may miss

The run ends with a warning for each kind of variant an annotation source may miss or read at the wrong place: what it is, with one example; why; and what to do. Every variant is still in the export:

altar run: warning: annotation sources may miss 1 indel: chr7:117559591:TCTT:T
  It was trimmed but not left-aligned, because [variants] names no reference genome; gnomAD and ClinVar store indels left-aligned.
  Set [variants] reference or reference_fasta to left-align it. Every variant is still in the export.
Reason What to do
indel_not_left_aligned Set [variants] reference_fasta.
reference_mismatch Check the REF alleles, and that [run] genome_build is the cohort's build.
contig_not_in_reference Use a FASTA of the cohort's build that holds every contig the cohort uses.
contig_not_primary Nothing, if expected: the published datasets cover chr1–22, X, Y and M only.

The run summary's potential_misses holds the same counts. Annotate variants with evidence explains each kind.

VCF alleles were skipped

altar run: warning: skipped 3 VCF ALT alleles that name no sequence a variant key can hold: 2 symbolic (<...>), 1 spanning deletion (*) (e.g. chr1:100:A:<DEL>)
  The export does not have them.

Symbolic alleles, breakends, *, ., an ALT equal to REF and IUPAC codes have no sequence to score or look up. The run summary's dropped_alleles counts them by reason.

An image or Docker was not found

altar run: warning: [models.legnet-wtc11] image ... is not on this machine, and its registry did not return it (...). The run will fail when it starts the container unless the image can be pulled by then; check the image name and digest
altar run: warning: [backend] the local backend runs containers with docker, which is not on PATH

The run continues, since the registry may be out of reach only for now. Pull the image, or install Docker, before the run reaches that model.

Other warnings

Warning Meaning
[export] format = "csv" is a deprecated name for "tsv" Write format = "tsv", or leave format out for Parquet.
the TSV holds ... as JSON text, which can exceed a spreadsheet's cell limit Leave [export] format out to keep list and record columns as lists in Parquet.

The run scored everything again

A model's line says how many variants were already scored:

[models.legnet-wtc11] 5 candidates, 1 set aside (1 deletion; listed in legnet-demo/staging/jobs/legnet-demo/variants/legnet-wtc11/ineligible.tsv), 4 already scored, 0 scored in 0 batches, in 0.0s

When a rerun scores variants you expected to be reused, check what changed:

  • the model key, the <name> of [models.<name>], under which the store keeps the scores;
  • a setting that changes the scores, which starts a new lineage beside the old one;
  • the reference FASTA, for a model that reads it;
  • the cache, for a spec without [store]: another starting directory, [run] cache_dir or ALTAR_CACHE_DIR finds another scores.sqlite;
  • the [store] path, or the directory the command starts in when the path is relative;
  • a [store] without a path, which keeps scores in memory for one run only;
  • a [detail_store] that moved, for a model with detail tables.

Keep scores in a SQLite file shows a store that later runs reuse.

A variant counts as scored when the store holds its row for the model's key and settings, and its detail rows and artifacts when the model has them. Score reuse and provenance explains when a score is reused, and Tell exports apart shows which settings an export's scores came from.

A model set variants aside

The same line counts the variants the model set aside, by reason, and names the file that lists them. A model scores only the variant classes and allele lengths its binding accepts; altar models <type> lists them. A variant whose REF does not match the reference, or whose contig the reference lacks, is set aside by every model.

Get more help

Run the command again with -v and keep its standard error. Keep the run summary on standard output, or _run_summary.json in the export directory. Get help says where to report a problem.

Next

  • Tutorials builds one analysis across five notebooks.
  • The catalog lists what each model and annotation source needs and accepts.