Command-line reference¶
altar is the console script the altar package installs. Each command below is a subcommand of it.
| Command | What it does | Label |
|---|---|---|
altar run |
Read a cohort, score and annotate it, and export one table, as a run spec describes. | Stable; --validate is Provisional |
altar spec-schema |
Print the JSON Schema of a run spec. | Stable |
altar prepare |
Run a model's preparation workflow before scoring. | Provisional |
altar interpret |
Run a spec's interpretation sections, or read what they stored. | Provisional |
altar models |
List the installed models, or show one. | Stable |
altar interpretations |
List the interpretation methods each installed model offers. | Provisional |
altar sources |
List the installed annotation sources, or show one. | Stable |
altar pull |
Copy a source's published dataset into storage you own. | Provisional; --bigquery is Preview |
altar resources |
List the reference genomes and model weights Altar can fetch, show one, or fetch one. | Provisional |
Maturity labels says what each label means.
Every command takes -h or --help. Results go to standard output. Progress, warnings and errors go to standard
error, written so the two kinds are easy to tell apart:
- A progress line has no prefix. It starts with the run spec section it belongs to, such as
[variants],[models.<name>],[sources.<name>],[export]or[interpretations.<name>]. The--validateplan prints the same way. The last line of a run,done in ..., has no section. - A problem starts with the command's name and its kind:
altar run: warning: ...for something the run went on past, andaltar run: error: ...for what stopped it. A longer problem continues on lines indented two spaces.
[variants] read 5 variants in 0.0s
done in 3.8s
altar run: warning: annotation sources may miss 1 indel: chr7:117559590:ATCT:A
It was trimmed but not left-aligned, because [variants] names no reference genome; gnomAD and ClinVar store indels left-aligned.
Set [variants] reference or reference_fasta to left-align it. Every variant is still in the export.
Warnings from the libraries a command uses, and from Python itself, are printed the same way. -q prints problems
only. -v adds debug lines, the full traceback of an error, a Python warning's category and the file it came from,
and the time, as HH:MM:SS, at the start of every line. The run spec reference documents the file
that run, prepare and interpret read.
altar run¶
Reads the cohort in [variants], scores it with each [models.<name>], annotates it with each
[sources.<name>], and writes one table with a row per variant. A spec without models only annotates.
| Argument | Meaning | Default |
|---|---|---|
spec |
The run spec, a .toml or .json file. |
Required. |
--validate |
Provisional. Check the spec and stop before reading the cohort. It builds the sources, which may open their published copies over the network, checks every genome build, parses each model's configuration, builds the backend, and looks up each image. It reads no variant and writes no file. | Off. |
-q, --quiet |
Print only warnings and errors on standard error. | Off. |
-v, --verbose |
Print debug lines, the full traceback when the run stops, and the time on every line. | Off. |
-q and -v cannot be combined.
| Stream or file | Contents |
|---|---|
| Standard output | The run summary as JSON: job_id, genome_build, candidate_variants, variant_count, models, export_uri, sources, potential_misses, dropped_alleles, provenance and sections. With --validate, the plan instead: job_id, genome_build, work_dir, cache_dir, variants, reference_fasta, models, sources, store, export_format, export_uri, compute and sections. sections lists each section whose resources is a list: its name, count, and its models' keys. |
| Standard error | One line per stage, ending with any skipped VCF alleles and the variants a lookup may miss. With --validate, a few lines that say what the run would do. A run that stops prints one altar run: error: line, followed by one indented line per problem for a spec that does not parse. |
<work_dir>/variants/<job_id>.candidates.tsv |
The cohort, normalized. work_dir is <job_id> by default. |
<work_dir>/staging/jobs/<job_id>/ |
The variant batches staged for the models' tasks, and each model's ineligible-variant report. |
<work_dir>/results/jobs/<job_id>/ |
The files the models' container tasks wrote. |
| The export directory | part-000000000000.parquet, or part-000000000000.tsv.gz with _provenance.json beside it, and _run_summary.json, a copy of standard output. [export] destination names it; the default is <work_dir>/export. |
[store], [detail_store], [artifact_store] |
The scores, detail rows and artifact files, where those sections say. Without [store], a spec with models keeps its scores in <cache_dir>/scores.sqlite, and the storage artifact store keeps its files in <cache_dir>/artifacts/. cache_dir is $ALTAR_CACHE_DIR, else .altar, and is created only when something is written there. |
<cache_dir>/backend-data/ |
The local backend's directory mounted into containers, unless [backend] options name a data_root. |
Read a run's results describes the export and the summary.
altar spec-schema¶
Prints the JSON Schema of a run spec on standard output: every section, its keys and their types. It takes no
argument and reads no file. The schema covers the spec's own keys. The keys inside options, configuration and
run_inputs belong to each adapter, model and source, which altar models and altar sources describe.
altar prepare¶
Runs the preparation workflow of the selected models, such as ChromBPNet's per-fold
calibration, and publishes their files under <cache_dir>/staging, where altar run reads them. Every invocation
runs the workflow again. altar prepare is Provisional.
It reads [run], [models.<name>], [backend], [storage], [connections.<name>] and the reference in
[variants]. It ignores the other sections and needs no variants.path and no [store]. An unknown top-level
section is refused.
| Argument | Meaning | Default |
|---|---|---|
spec |
A preparation spec or a full run spec, .toml or .json. |
Required. |
--model |
A model key, the <name> of a [models.<name>] section, to prepare. Repeat it to select several. |
Every model in the spec, each of which must offer preparation. |
--validate |
Check the selected models' configurations, plans and backend, and look up each image, without hashing the reference, running a task or writing a file. | Off. |
-q, --quiet |
Print only warnings and errors on standard error. | Off. |
-v, --verbose |
Print debug lines, and the full traceback when preparation stops. | Off. |
| Stream or file | Contents |
|---|---|
| Standard output | JSON. With --validate: for each model key, its plugin_id, its stages and the output_uris it would publish. Otherwise: job_id and models, each with name, plugin_id, completed_stages and output_uris. |
| Standard error | Progress. With --validate, the stages each model would run and where its files would go. |
<cache_dir>/preparation-attempts/<name>-<hex>/ |
Each attempt's collected outputs. Kept for diagnosis. |
<cache_dir>/staging/models/<name>/ |
The published files. |
<cache_dir>/preparation-state/<name>.json |
The record of a complete publication, which altar run checks against the model's configuration. |
<cache_dir>/preparation-state/<name>.incomplete |
Present while a publication is in progress. While it exists, altar run refuses the model. |
Prepare model resources walks through a preparation and its recovery.
altar interpret¶
Runs the spec's [interpretations.<name>] sections on its [backend] and keeps every product in the
interpretation store. Variants whose products are stored are not computed again. Running the same command again
resumes an interrupted execution and retries variants a failed one left without products. It scores nothing.
altar interpret is Provisional.
| Argument | Meaning | Default |
|---|---|---|
spec |
A run spec with [interpretations.<name>] sections. |
Required. |
--section |
Run only this [interpretations.<name>] section. Repeat it to select several. |
Every section. |
--validate |
Check the sections, models, methods, targets and backend, print what would run, and stop. | Off. |
--read |
Run nothing. Print each section's stored products as JSON, and list them in products.tsv. |
Off. |
--variant |
With --read, read only this variant, as chr:pos:ref:alt. Repeat it to read several. Without --read it is a usage error. |
Every stored variant. |
-q, --quiet |
Print only warnings and errors on standard error. | Off. |
-v, --verbose |
Print debug lines, and the full traceback when the command stops. | Off. |
--validate and --read cannot be combined, nor can -q and -v.
| Stream or file | Contents |
|---|---|
| Standard output | JSON. With --validate: for each section, its model, plugin_id, methods, target and variants ("cohort" or a count). Otherwise: job_id, genome_build, store, products_uri, operations and failed, plus products with --read. |
| Standard error | One line per operation. With --validate, one line per section saying what it would run. |
<work_dir>/interpretation-export/products.tsv |
One row per stored product file and variant. A --read limited by --variant does not write it. |
<work_dir>/interpretation-export/operations/<hex>.json |
Each operation the rows name. |
<work_dir>/interpretation-export/_interpretation_summary.json |
A copy of standard output. |
<cache_dir>/interpretation-runs/<run_id>.json |
Each execution's progress, which a rerun resumes from, with the files it staged and collected under <cache_dir>/staging/interpretations/ and <cache_dir>/results/interpretations/. |
| The interpretation store | The product files and their index; see interpretation_store. |
altar interpret my-run.toml --validate
altar interpret my-run.toml --section contributions
altar interpret my-run.toml --read --variant chr6:136274285:A:G
Interpret variants shows how to read the products.
altar models¶
Lists the installed models, grouped by the distribution that ships each one. Named, it shows one model in full:
a [models.<name>] section to paste, every configuration field, the genome builds and variant classes it
accepts, its columns, detail tables and artifacts, its default image, the compute its tasks ask for, the command
that fetches its weights, whether it needs altar prepare, its interpretation methods, and its documentation page.
It loads no weights, builds nothing and pulls no image.
| Argument | Meaning | Default |
|---|---|---|
name |
The model type to show, such as legnet. Matched without regard to case. |
List every model. |
--json |
Print the listing, or the one model, as JSON. | Text. |
| Stream | Contents |
|---|---|
| Standard output | The listing or the model, as text or JSON. The JSON listing is {"models": [...], "failures": [...]}. |
| Standard error | One line for each installed model that could not be loaded, or the closest installed names for a name that is not installed. |
altar interpretations¶
Lists the installed models that interpret: each method, its version, the targets it explains, the method it runs
first, and the product files it writes. Named, it also describes each product file. It builds and runs nothing.
altar interpretations is Provisional.
| Argument | Meaning | Default |
|---|---|---|
name |
The model type to show, such as chrombpnet. Matched without regard to case. |
Every model that interprets. |
--json |
Print the listing as JSON. | Text. |
| Stream | Contents |
|---|---|
| Standard output | The listing as text, or {"models": [...], "failures": [...]} as JSON. |
| Standard error | One line for each model that could not be loaded, or why a named model was refused. |
altar sources¶
Lists the installed annotation sources: the ready-to-use ones first, then the generic ones, then the
variant-gene link sources. Named, it shows one source in full: its columns, a [sources.<name>] section to paste,
its options, its published data and the command that builds its data. It builds no source and opens no network
connection.
| Argument | Meaning | Default |
|---|---|---|
name |
The source to show, such as gnomad. Matched without regard to case. A link source's name, such as e2g_atlas, shows how a run reads its links. |
List every source. |
--json |
Print the listing, or the one source, as JSON. | Text. |
| Stream | Contents |
|---|---|
| Standard output | The listing or the source, as text or JSON. The JSON listing is {"sources": [...], "link_sources": [...], "failures": [...]}. |
| Standard error | One line for each source that could not be loaded, or the closest installed names for a name that is not installed. |
altar pull¶
altar pull [-h] (--to PATH_OR_URI | --bigquery PROJECT.DATASET.TABLE) [--from LOCATION] [--project PROJECT]
[--impersonate-service-account EMAIL] [--source-project PROJECT] [--source-uri GS_URI]
[--overwrite] [-q]
source
Copies the dataset a source's publisher hosts into storage you own: its Parquet files to a path or URI, or loaded
into a BigQuery table. The published files are read anonymously. The destination is reached through Application
Default Credentials. A copy to --to resumes: rerun the same command and it skips the files already there.
altar pull is Provisional. The BigQuery load, --bigquery, is Preview: it has been tested only
against a fake BigQuery client, not against live BigQuery.
| Argument | Meaning | Default |
|---|---|---|
source |
The annotation source whose published dataset to copy, such as gnomad. altar sources lists the ones that publish. |
Required. |
--to |
Copy the Parquet files to this local path or URI, such as gs://bucket/dir. |
One of --to and --bigquery is required. |
--bigquery |
Load the files into this BigQuery table, project.dataset.table, with one load job. Refused for a source that reads no BigQuery table. |
One of --to and --bigquery is required. |
--from |
Read the published files from this named location, such as hf. |
The dataset's primary location. |
--project |
The Google Cloud project to bill, and to run the BigQuery load job in. | The project of the ambient credentials. |
--impersonate-service-account |
Write or load as this service account. | Your own credentials. |
--source-project |
Bill reads of the published files to this project, for a requester-pays bucket. | Read anonymously. |
--source-uri |
With --bigquery only: load from this gs:// copy instead of the published one. |
The published files. |
--overwrite |
With --to, copy every file again, replacing files of the same name, even beside files that are not the dataset's. With --bigquery, replace the table's rows. |
Resume, and refuse a destination holding other files or a table that has rows. |
-q, --quiet |
Print no progress on standard error. | Off. |
| Stream or file | Contents |
|---|---|
| Standard output | JSON. For --to: destination, source_uri, files, bytes, skipped and identity. For --bigquery: table, source_uri, job_id, rows, write_disposition, clustering_fields, identity and labels. |
| Standard error | The size before it starts, one line per file copied with an estimate of the time left, and the outcome. A BigQuery load reports only its start and its end. A refusal is one line that says what to do next. |
| The destination | The dataset's Parquet files, under their partition directories, or the loaded table. |
altar pull gnomad --to gs://my-bucket/gnomad-4.1.1-genomes
altar pull gnomad --from hf --to ./gnomad-4.1.1-genomes
altar pull clinvar --bigquery my-project.reference.clinvar --project my-project
Choose where data is read from describes the copy, the load and the release checks.
altar resources¶
altar resources [REGISTRY [--where KEY=VALUE]... [--search WORDS] | REGISTRY/ID | fetch [NAME...] | check URI] [--spec SPEC] [--cache-dir DIR] [--refresh] [--keep-archives] [--jobs N] [--yes] [--json] [-q]
Lists the registries a run spec can name: the ones installed packages ship, such as genome and legnet, those
ALTAR_REGISTRIES names (name=uri,name=uri), and with --spec the spec's [registries]. Each row gives the
registry's name, how many entries it lists, where it comes from and its description. Named, a registry lists its
entries, led by each entry's name, with the id to copy, the model type, the genome it was trained on, its size and
whether it is fetched; --where and --search filter them. A registry of genomes that say who publishes them
lists each genome's id, alias, build, authority, accession and product instead, and a line on what sets each one
apart. Named as REGISTRY/ID, an entry is shown in full: its files with their digests and addresses, the run-spec
lines that name it, its training genome, metadata, licence and citation, and a genome's aliases and publisher. The
registry is a name's first segment and the id the rest, which may hold / itself, such as
genome/grch38/ncbi/no-alt-analysis-set. A genome needs no genome/: its id, such as grch38/ucsc/hg38, or its
alias, such as hg38, names it, as does any name whose first segment names no registry. Names are matched
ignoring case. A registry
on this machine is read in place; any other index is read once and kept in the cache, and --refresh reads it
again. altar resources is Provisional.
altar resources fetch NAME... fetches each entry: its entry document, checked against the digest the index
gives, and every file it names, each into <cache>/resources/sha256/<hex> and checked against its SHA-256 digest.
A file shared by two entries is stored once. Members of one archive are extracted from one download of it, which
is deleted afterwards unless --keep-archives. A file whose digest differs is deleted, and the command stops,
naming the entry, the address and both digests. Each verified file gets a verification record beside it, so a
rerun skips it and a run accepts it without hashing it again. A genome also gets its .fai index, checked against
its contigs. A run spec then names the entry: [variants] reference = "hg38" or [models.<name>] resources =
"legnet/wtc11".
altar resources fetch --spec SPEC, with no entry named, fetches every entry the spec names: its [variants]
reference and each model's resources, every entry of a list too. It first prints how many entries there are,
how many are already fetched, and the size still to fetch. Above 20 GB, or ALTAR_FETCH_CONFIRM_BYTES, or when
an entry's registry gives no size, it asks you to confirm; --yes confirms, and without a terminal it stops with
the size or how many entries have none. Entries are fetched up to
--jobs at once; fetches that run at once, even of one file, keep the cache whole. The first failure, or
Ctrl-C, stops the others at their next read, which waits at most 60 seconds on a stalled connection, and keeps every
file already verified.
A registry that is not on your disk can only make fetch read files at public https://, gs://, s3:// or
hf:// addresses. It cannot point a fetch at a file on your machine, at plain http://, or at a host on your own
network; redirects are checked the same way. Cloud files are read without your credentials unless the registry
reads through a connection, and a download stops once it is larger than the entry says, or than 64 GB when the
entry gives no size. Find and fetch models says how to allow an intranet
registry.
altar resources check URI checks a registry you publish: the index and every entry it names, against the
format and the rules of Publish a registry. It reads no other file.
| Argument | Meaning | Default |
|---|---|---|
name |
A registry, such as legnet, an entry, REGISTRY/ID or a genome id; fetch followed by one or more entries, or by none with --spec; or check followed by the URI of an index. |
List the registries. |
--spec |
A run spec whose [registries] to add, with the [connections] they read through, and whose [run] cache_dir to use. fetch with no entry fetches every entry it names, reading its [variants] reference and each model's resources; its other sections are not read. |
None. |
--cache-dir |
The cache to list or fetch into. A run reads entries from its [run] cache_dir, so give the same directory. |
The spec's [run] cache_dir with --spec; else ALTAR_CACHE_DIR, or ./.altar when it is unset, as for a run. |
--where |
With a registry, list only entries whose KEY matches VALUE ignoring case: name, id, type, genome_build, or a facet (assay, cell_types, organs, systems, life_stages, sex, organism, project). An ontology term matches its id or its name. Repeat it to require every one. |
Every entry. |
--search |
With a registry, list only entries with a word of their name, id or facets that starts with each word given. | Every entry. |
--refresh |
Read each registry's index again from where it is published. | Read a remote index once, when the cache has none. |
--keep-archives |
With fetch, keep the archives entries take members from, under <cache>/resources/archives/. |
Delete them. |
--jobs |
With fetch, fetch up to this many entries at once. Progress lines then start with the entry's name. |
4 |
--yes |
With fetch --spec, fetch without asking, however much it downloads. |
Ask above 20 GB, or for an entry of unknown size. |
--json |
Print the listing, the entry, the fetch summary, or the check's problems as JSON. | Text. |
-q, --quiet |
With fetch, print no progress on standard error. |
Off. |
| Stream or file | Contents |
|---|---|
| Standard output | The listing or the entry, as text or JSON. The JSON listing of registries is {"registries": [...], "failures": [...]}, and of entries {"registry", "total", "entries": [...]}. For fetch, JSON per entry: name, type, entry_sha256, files_fetched, files_present, bytes_fetched, and files, where the entry names each file and its path in the cache. With --spec and no entry, a list of those for the entries it fetched, empty when every one was fetched already. For check, one problem per line, or {"uri", "problems"} as JSON. |
| Standard error | For fetch --spec, first the count of entries, how many are fetched and the size still to fetch. For fetch, the size before it starts, one line per file verified, a line every 30 seconds while a large file or archive downloads, and the run-spec lines that name the entry. A refusal is one line that says what to do next. |
<cache>/resources/sha256/ |
Every fetched file under its digest, each with a .sha256-verified record beside it, and a genome's .fai beside its FASTA. |
<cache>/registries/<name>/ |
A remote registry's cached index, the entry documents fetched from it, and a record of each fetch, fetched/<id>.json, with each / of the id written %2F. |
altar resources
altar resources legnet
altar resources lab --where assay=dnase-seq --search k562 --spec spec.toml
altar resources genome/grch38/ucsc/hg38
altar resources grch38/ncbi/no-alt-analysis-set
altar resources fetch hg38 legnet/wtc11
altar resources fetch legnet/wtc11 --cache-dir /data/altar-cache
altar resources fetch --spec score.toml --jobs 8 --yes
altar resources check https://example.org/registry/registry.json
Find and fetch models and genomes shows how a run spec names a fetched entry.
Exit statuses¶
| Status | Meaning |
|---|---|
0 |
The command succeeded. |
1 |
The command failed. For run, prepare and interpret, something the checks could not foresee went wrong, such as a container task that failed or a lost connection. Its error: line is followed by Run again with -v to see where it stopped. Tasks still open after [backend] timeout also exit 1, with an error: line that says what to change. For interpret, also an operation that left variants without products. For models, sources and interpretations, the named plugin is installed but could not be loaded. For pull, an error it does not recognize, printed as a Python traceback. For resources fetch, a download that failed; rerun it to resume. |
2 |
The command refused its input. The spec cannot run as written, and the message names the section. An optional extra is missing, a file cannot be found or opened, or Google Cloud reports missing credentials, a missing permission or a missing project or dataset. For models, sources, interpretations, pull and resources, the name is not installed; for interpretations, also a model that declares no methods; for pull, any refusal it explains; for resources fetch, a file whose digest does not match; for resources check, a registry with a problem. Command-line usage errors also exit 2. |
130 |
Interrupted with Ctrl-C (run, prepare, interpret, pull, resources fetch). altar run writes no export. |
Fix a run that stops or warns lists the common messages and what to do.
Environment variables¶
| Variable | Read by | Meaning |
|---|---|---|
The name a spec gives in { provider = "env", name = "..." } |
altar run |
A model's credential, such as ALPHAGENOME_API_KEY. The spec holds only the name. An unset or empty variable stops the run before it reads the cohort. |
ALTAR_FETCH_MAX_BYTES |
altar resources fetch |
The most bytes one file or archive may take when its registry entry gives no size. Without it, 64 GB. |
ALTAR_FETCH_CONFIRM_BYTES |
altar resources fetch --spec |
The most bytes it downloads without asking you to confirm, or --yes. Without it, 20 GB. |
ALTAR_CACHE_DIR |
altar run, altar prepare, altar interpret, the local backend |
The cache when [run] cache_dir names none. Without either, .altar in the starting directory. The local backend mounts backend-data in it into each container when [backend] options give no data_root. |
ALTAR_K8S_NAMESPACE |
The kubernetes backend |
The namespace when options give no namespace. Without either, default. |
ALTAR_K8S_PVC |
The kubernetes backend |
The PersistentVolumeClaim mounted in every pod when options give no pvc_name. Without either, no volume is mounted. |
ALTAR_MODAL_APP |
The modal backend |
The Modal App when options give no app_name. Without either, altar-exec. |
ALTAR_MODAL_VOLUME |
The modal backend |
The Modal Volume when options give no volume_name. Without either, altar-data. |
ALTAR_VARIANTS_DATA_DIR |
The regions source |
The region data directory when its options give no data_dir. Without either, the installed package's data directory. |
Provider libraries read their own credentials, such as GOOGLE_APPLICATION_CREDENTIALS for Google Cloud's
Application Default Credentials.
Binding commands¶
Bindings install their own commands to fetch model weights or build a source's dataset. Each one's model or source
page documents it; <command> --help lists its flags. LegNet's weights and the reference genomes are fetched with
altar resources fetch instead.
| Command | Distribution | Documented on |
|---|---|---|
altar-borzoi-resources |
altar-borzoi |
Borzoi |
altar-cherimoya-resources |
altar-cherimoya |
Cherimoya |
altar-chrombpnet-resources |
altar-chrombpnet |
ChromBPNet |
altar-enformer-resources |
altar-enformer |
Enformer |
altar-popfreq-build |
altar-popfreq |
gnomAD |
altar-clinvar-build |
altar-clinvar |
ClinVar |
altar-conservation-build |
altar-conservation |
phyloP 100-way |
altar-opentargets-e2g-build |
altar-opentargets-e2g |
Open Targets E2G |
altar-alphamissense-build |
altar-alphamissense |
AlphaMissense |
altar-spliceai-build |
altar-spliceai |
SpliceAI |
altar-gpnstar-build |
altar-gpnstar |
GPN-Star |
altar-e2g-atlas-ingest |
altar-e2g-atlas |
ENCODE-rE2G and scE2G atlases |
Sei's weights are fetched with sei-resources, a command of the Sei runtime in a source checkout; see
Sei.