Interpret variants¶
An interpretation explains a model's prediction for a variant: which bases drive it and, for some methods, which
motifs those bases form. altar interpret runs the [interpretations.<name>] sections of a run spec, stores every
file the model writes, and lists each variant's files with the reference span each row covers.
Provisional
Interpretation is Provisional: the altar interpret and altar interpretations commands, the
[interpretations] and [interpretation_store] sections and the output files are expected to change.
See Maturity labels.
Before you start¶
- A model that interprets, ChromBPNet or Cherimoya, with its resources fetched.
- Docker on an amd64 machine; see Run tasks on a GPU or a cluster. Interpretation runs on its own
pinned images, which the backend pulls on first use. The methods ask for a GPU;
[backend] cpu_only = trueruns them on the CPU. - The reference FASTA the model scores with.
altar interpretations lists the methods of every installed model. ChromBPNet offers deepshap_counts and
finemo_hits, which needs a motif set. Cherimoya offers ism_counts:
$ altar interpretations
Interpretation methods explain a model's predictions for variants. To run one, add a section to a run
spec that names one of its models and the method, then run `altar interpret spec.toml`:
[interpretations.explain]
model = "<a [models.<name>] key of the spec>"
method = "ism_counts"
Run `altar interpretations <model>` to see a model's products.
CHERIMOYA (altar-cherimoya)
ism_counts 1.0.0: target counts
In-silico saturation mutagenesis of the count head over the central 400 bases of each allele,
averaged over the folds.
Writes ism_scores (application/x-npy), ism_sequences (application/x-npy).
CHROMBPNET (altar-chrombpnet)
deepshap_counts 2.0.0: target counts
DeepSHAP contributions of the counts head and predicted profiles, averaged over the folds.
Writes contribution_scores (application/x-npy), predicted_profiles (application/x-npy).
finemo_hits 1.0.0: target counts; runs deepshap_counts first
finemo motif hits on each allele's averaged contributions, and a per-variant summary table.
Writes motif_hits (text/tab-separated-values), variant_summary (text/tab-separated-values).
altar interpretations chrombpnet describes each product file.
Add an interpretation section¶
An [interpretations.<name>] section names a model of the spec and a method. The repository's
examples/interpretation spec,
chrombpnet_k562.toml, interprets three of the quickstart's chr6 SNVs with the K562 DNase ChromBPNet model on the
CPU. Its README says how to fetch the weights and the motif set. The spec, without its comments:
[run]
job_id = "k562-interpret"
genome_build = "hg38"
[variants]
path = "variants.tsv"
reference_fasta = "hg38.fa"
[models.k562]
type = "CHROMBPNET"
model_name = "K562 DNase (ENCSR000EOT)"
resource_manifest = "resources/chrombpnet.resources.json"
[backend]
type = "local"
cpu_only = true
poll_interval_s = 5.0
[interpretations.contributions]
model = "k562"
method = "deepshap_counts"
[interpretations.motifs]
model = "k562"
method = "finemo_hits"
[interpretations.motifs.configuration]
motifs = { uri = "motifs/counts/tfmodisco.raw_output.counts.ENCSR000EOT.hd5", digest = "sha256:191f14518e2ac960d5de6b49e6babcbf18c1f70601551f7adac45afb83376c31" }
modelnames a model key. The interpretation uses that model's type, configuration, resources and run inputs, and stores its products under the key.methodnames one method, andmethodsseveral, each run as its own request. A method runs every method it requires, so themotifssection storesdeepshap_countsproducts too.configurationadds settings only interpretation reads, such as the motif setfinemo_hitsneeds, or ChromBPNet'sreference_seed.variants, not used here, lists variant IDs to interpret in place of the cohort. The spec still needs[variants] path.
ChromBPNet's deepshap_counts compares each allele with shuffled copies of its sequence, drawn from a random
stream. reference_seed sets that stream's seed. It is a whole number from 0, and 0 is the default. Another seed
gives other contributions, stored beside the seed-0 ones, and never changes a score:
[interpretations.contributions-seed-1]
model = "k562"
method = "deepshap_counts"
[interpretations.contributions-seed-1.configuration]
reference_seed = 1
A spec whose only work is interpretation needs a [backend] and no [store]. altar run ignores these
sections, and altar interpret scores nothing. The run spec reference
lists every key.
Run it¶
Check the sections first. --validate checks the models, methods, targets and backend, and runs nothing:
$ altar interpret chrombpnet_k562.toml --validate
[interpretations.contributions] would run deepshap_counts (counts) with [models.k562] on cohort variants
[interpretations.motifs] would run finemo_hits (counts) with [models.k562] on cohort variants
{
"contributions": {
"model": "k562",
"plugin_id": "org.kundajelab.altar.model.chrombpnet",
"methods": [
"deepshap_counts"
],
"target": "counts",
"variants": "cohort"
},
"motifs": {
"model": "k562",
"plugin_id": "org.kundajelab.altar.model.chrombpnet",
"methods": [
"finemo_hits"
],
"target": "counts",
"variants": "cohort"
}
}
Run every section, or one with --section. The command-line reference
lists every flag.
The command prints one line per request on standard error, and a JSON summary on standard output:
[interpretations.contributions] deepshap_counts (counts) of [models.k562]: 3 variants, 0 already stored, ran 3 as execution contributions-92a11cb2e4fc, committed deepshap_counts, in 2m39s
Before it starts a container, the command refuses a variant the model cannot interpret, such as one whose input window runs off its contig or whose REF is not the reference's.
Variants whose products are stored are not computed again. Running the spec again computes nothing, and a cohort with new variants runs only the new ones.
Read the products¶
The products go to an interpretation store, by default under .altar/interpretations, in the cache a later
run reads them from. altar interpret lists them in interpretation-export/ in the run's directory,
k562-interpret/ here, named after job_id:
products.tsvhas one row per stored product file and variant: the file'suri, the variant'srowin it, the reference span the row covers (chrom, 0-basedstart,end, andanchor), and the IDs of the request and execution that produced it.operations/<operation hex>.jsondescribes each request the rows name: the model resources and genome by digest, the images, the method's settings, and its input variants._interpretation_summary.jsonis the JSON the command printed.
--read runs nothing and prints the stored products. --variant limits it to some variants:
Each product in the printed products list looks like this:
{
"section": "contributions",
"model_id": "k562",
"method": "deepshap_counts",
"variant_id": "chr6:136274285:A:G",
"product": "contribution_scores",
"shard": "allele=ref",
"run_id": "contributions-92a11cb2e4fc",
"operation_id": "sha256:92a11cb2e4fce6c26f98ca8bdca4ee92391f069ec635c04ed6c82b65668c76f1",
"interpretation_id": "sha256:aff56a1aa5c8ebafd916fffd625080ccb3dfaa7b60e26c458f036940d4900934",
"uri": ".altar/interpretations/k562/contribution_scores/aff56a1a…/76a2b4a6….npy",
"row": 2,
"chrom": "chr6",
"start": 136273227,
"end": 136275341,
"anchor": 1057,
"media_type": "application/x-npy",
"digest": "sha256:76a2b4a6df032f01491940852a31e9fc115598b9e5c03d2147b7cb0fc5228670"
}
uri is the stored file and row is the variant's row in it. start and end are the 0-based, half-open
reference span the row covers, and start + anchor is the variant's 0-based position. shard names the part
of the output the file holds, such as allele=ref for the reference allele. Element i of a ChromBPNet
contribution_scores row is the contribution of reference base start + i. Each model's page gives the shape of
its products and how bases after an indel map.
products.tsv holds the same fields, one product per line. Load a variant's contributions with NumPy:
import numpy as np
import pandas as pd
products = pd.read_csv("k562-interpret/interpretation-export/products.tsv", sep="\t")
match = products[(products["variant_id"] == "chr6:136274285:A:G") & (products["product"] == "contribution_scores")]
for product in match.itertuples():
scores = np.load(product.uri)[product.row]
print(product.shard, product.chrom, product.start, scores[product.anchor])
Run and read interpretations reads the products as Python objects and runs a request on a backend you build.
Check the result¶
products.tsvhas a row per product file for each variant you asked for. Fordeepshap_countsthat is four per variant: each of its two products has one file per allele.- The summary lists, for each request, the methods it
committedand the variants stillmissingproducts.missingis empty when every variant was interpreted.
If it fails¶
- A method that fails ends the command with exit status 1. The methods before it in the chain stay stored, so a
failed
finemo_hitskeeps itsdeepshap_countsproducts. - Run the same command again after a failure or an interruption. An execution stopped while its tasks ran runs
again. One stopped after its files were collected stores them without running. A failed execution is not
rerun as is: the next run retries its missing variants as a new execution, labelled
<run_id>-attempt2.
Each execution records its progress in .altar/interpretation-runs/, beside the files it resumes from. Keep
them with the interpretation store: deleting them while the store remains makes a rerun of a partly failed
execution stop. Deleting the run's own directory, k562-interpret/, loses nothing a rerun needs.
Fix a run that stops or warns covers spec errors.
Next¶
- Fix a run that stops or warns covers the messages of
altar run,altar prepareandaltar interpret. - ChromBPNet and Cherimoya describe each product's shape, the settings in each interpretation ID, and how replays compare across devices.
- Run and read interpretations covers the Python API, and running each step from your own program.