Skip to content

Read a run's results

altar run writes one table with a row per variant, a summary of what the run did, and a record of which data and settings each column came from. A model's detail tables and artifacts stay in the run's stores. A separate command adds the results to the input VCF as INFO fields.

Before you start

  • A finished run. The examples use the quickstart's runs, whose files are in a directory named after each job_id: first-annotation from Annotate variants and first-scores from Score variants.
  • pandas or pyarrow to read Parquet. Both come with Altar.

Read the export

The export directory is <work_dir>/export unless [export] destination names another. It holds the table, a list of the table's columns and a copy of the run summary:

$ ls first-annotation/export
_columns.tsv  _run_summary.json  part-000000000000.parquet

Read the directory with pandas or pyarrow. Both skip _columns.tsv and _run_summary.json, because their names start with an underscore:

import pandas as pd

table = pd.read_parquet("first-annotation/export")

The table has one row per variant of the cohort, sorted by variant_id. Its columns are:

  • variant_id, the variant as chr:pos:ref:alt;
  • prioritized, true when a model or a prioritizing source flags the variant;
  • each source's columns, such as gnomad_af and clinvar_classification. Each source's page lists them;
  • each model's score columns and its own prioritized column, under a prefix of the model key and a short hash, such as legnet_wtc11__60a71072bf29_activity_delta. A store that keeps lineages adds the model's score_lineage_id, which names the set of score-changing settings the row's scores came from;
  • prioritized_model_ids and prioritized_source_names, the models and sources that flagged the variant.

An empty value means the source holds no record of the variant, or the model did not score it. It is not a zero. What "prioritized" means describes the flags.

Find a model's columns

_columns.tsv lists the table's columns in order, one row per column. Its columns are:

Column What it holds
column The column's name in the table.
kind variant_key (variant_id), annotation (a source's column), score (a model's score), prioritization (a prioritized flag or one of the two lists of what flagged the variant), lineage (a model's score_lineage_id), or display (a column a BigQuery display rider adds).
field The value's own name: a score's name, such as activity_delta, which every model of one binding shares; a source's column name; or the column's name otherwise.
dtype, repeated The value's type, float, int, bool, str, json or struct, and true when the column holds a list of it.
label A score's or a source column's display label.
source The source of an annotation column.
model_id, model_name For a model's column: the model key and its display name.
plugin_id, plugin_version For a model's column: the binding that scored it, and its release.
result_schema_version, result_schema_hash For a model's column: the version and hash of the binding's score columns.

A model column's row, from the scores run:

column      legnet_wtc11__60a71072bf29_activity_delta
kind        score
field       activity_delta
dtype       float
model_id    legnet-wtc11
plugin_id   org.kundajelab.altar.model.legnet
…

Select one model's scores by its key, or one score across every model by its name:

import pandas as pd

columns = pd.read_csv("first-scores/export/_columns.tsv", sep="\t")
table = pd.read_parquet("first-scores/export")

legnet = columns.query("model_id == 'legnet-wtc11' and kind == 'score'")["column"]
deltas = columns.query("field == 'activity_delta'")[["column", "model_id"]]
scores = table[["variant_id", *legnet]]

The list depends only on the table's columns. In Python, altar.results.wide_column_manifest returns it for a table you write yourself; see Export results.

Choose the format

[export] format = "tsv" writes gzipped, tab-separated text with a header row, part-000000000000.tsv.gz, in place of Parquet:

[export]
format = "tsv"

Parquet keeps list and record columns, such as ClinVar's conditions and genes, as lists. TSV writes them as JSON text, and the run warns that one such cell can exceed a spreadsheet's cell limit:

altar run: warning: the TSV holds clinvar_conditions, clinvar_conflicting_classifications, clinvar_molecular_consequences, clinvar_origins, clinvar_genes as JSON text, which can exceed a spreadsheet's cell limit; leave [export] format out for Parquet, which keeps them as lists

[export] prioritized_only = true keeps only the prioritized rows. The run spec reference lists the export keys.

Long format for many models

A run with hundreds of models gives a table with thousands of columns. [export] layout = "long" writes the same values as a few narrow tables, with a row per variant and model in place of a block of columns per model:

[export]
layout = "long"

Use it when a run has many models, or when you load the results into a database or a data frame and filter by model. Keep the default wide table for a few models, a spreadsheet, or an augmented VCF, which reads the wide table only. The long format is provisional.

The export directory then holds one directory per table, each with one part, part-000000000000.parquet, or part-000000000000.tsv.gz with format = "tsv":

$ ls first-scores/export first-scores/export/scores
first-scores/export:
_provenance.json  _run_summary.json  models  scores  variants

first-scores/export/scores:
legnet_v0_0_1__15afe8e41575
Table One row per Columns
variants variant variant_id, prioritized, each source's columns, prioritized_model_ids and prioritized_source_names: the wide table's columns other than the models'
models model model_id (the model key), model_name, column_prefix (the prefix of its columns in the wide table), plugin_id, plugin_version, result_schema_version, result_schema_hash, table (the score table its rows are in), lineage_id (the lineage the run requested for it), and registry, registry_id and registry_entry_sha256 for a model named by a registry entry
scores/<table> variant and model variant_id, model_id, score_lineage_id on a store that keeps lineages, the binding's score columns under their own names, such as activity_delta, and the model's prioritized

Models of one binding and one version of its score columns share a score table, so 500 ChromBPNet models write one table with a logfc column. The table's name is the binding's name, its score columns' version and a short hash, such as chrombpnet_v0_0_1__6b809b253c84, and is the same in every run. A model that did not score a variant has no row for it. Every other value is the one the wide table holds. The provenance is in _provenance.json for both formats.

Join a score table to the models and the variants on model_id and variant_id:

import pandas as pd

export = "first-scores/export"
models = pd.read_parquet(f"{export}/models")
variants = pd.read_parquet(f"{export}/variants")

table = models.loc[models["plugin_id"] == "org.kundajelab.altar.model.legnet", "table"].iloc[0]
scores = pd.read_parquet(f"{export}/{table}")
legnet = scores.merge(models[["model_id", "model_name"]], on="model_id").merge(variants, on="variant_id")

Turn one score into a variant-by-model table:

deltas = scores.pivot(index="variant_id", columns="model_id", values="activity_delta")

Or query the files in place with DuckDB:

SELECT s.variant_id, m.model_name, s.activity_delta, v.gnomad_af
FROM 'first-scores/export/scores/legnet_v0_0_1__15afe8e41575/part-*.parquet' AS s
JOIN 'first-scores/export/models/part-*.parquet' AS m USING (model_id)
JOIN 'first-scores/export/variants/part-*.parquet' AS v USING (variant_id)
WHERE m.model_id = 'legnet-wtc11' AND s.prioritized;

Detail tables and artifacts stay in their stores in either layout; see Read details and artifacts. altar.results.write_long_export writes the same tables from Python.

Read its provenance

altar run prints the run summary as JSON on standard output, and writes the same JSON to _run_summary.json in a local export directory. For the annotation run, abbreviated:

{
  "job_id": "first-annotation",
  "genome_build": "hg38",
  "candidate_variants": "first-annotation/variants/first-annotation.candidates.tsv",
  "variant_count": 5,
  "models": [],
  "export_uri": "first-annotation/export",
  "sources": [
    {
      "name": "gnomad",
      "section": "sources.gnomad",
      "identity": {
        "source_id": "org.kundajelab.altar.annotation.gnomad",
        "schema_version": "1.0.0",
        "release": "4.1.1-genomes",
        "genome_build": "hg38"
      }
    },
    ...
  ],
  "potential_misses": [
    {
      "reason": "indel_not_left_aligned",
      "count": 1,
      "example": "chr7:117559590:ATCT:A",
      "message": "1 indel was only trimmed, not left-aligned, ..."
    }
  ],
  "dropped_alleles": {},
  "provenance": {...},
  "export_layout": "wide",
  "column_manifest_uri": "first-annotation/export/_columns.tsv",
  "export_tables": {}
}
  • sources names the release and genome build each source served.
  • potential_misses lists the variants a lookup may miss, and dropped_alleles counts the VCF alleles the run skipped, by reason.
  • models has one entry per model: its candidate, set-aside, already-scored and scored counts, ineligible_reasons with the set-aside count by reason, and the path of its list of set-aside variants.
  • provenance records which data and settings each column came from. The export carries it too.
  • export_layout is wide or long. column_manifest_uri names a wide export's list of columns, and export_tables maps each table of a long export to its directory.

Read the provenance from the export

The export carries the summary's provenance, so a file shared on its own still says where its columns came from:

Export Where the provenance is
Parquet the file's key-value metadata, under the key altar.provenance
TSV _provenance.json, beside the table
Long format, either file format _provenance.json, beside the tables

Read it from a Parquet file with pyarrow:

import json

import pyarrow.parquet as pq

metadata = pq.read_metadata("first-scores/export/part-000000000000.parquet").metadata
provenance = json.loads(metadata[b"altar.provenance"])

provenance is {"format": 1, "models": {...}, "sources": {...}, "export": {...}}. export holds the layout, and the column_manifest, _columns.tsv, beside a wide export. sources maps each source to the release and build the summary records for it. models has one entry per model key. Its model_name is the spec's model_name display name, or the model key when the spec sets none. LegNet's entry, abbreviated and with the hashes shortened:

{
  "model_name": "legnet-wtc11",
  "column_prefix": "legnet_wtc11__60a71072bf29",
  "plugin_id": "org.kundajelab.altar.model.legnet",
  "requested": {
    "lineage_id": "sha256:9c41…",
    "plugin_version": "0.0.1",
    "configuration_hash": "sha256:51d0…",
    "runtimes": [
      {"name": "model_runtime", "kind": "container_image", "identity": "riyavsinha/altar-legnet@sha256:43a4…"}
    ],
    "genome_build": "hg38",
    "outputs": ["scores"],
    "…": "…"
  },
  "reuse": {"scope": "current", "lineages": [{"lineage_id": "sha256:9c41…", "step": "current"}], "…": "…"},
  "this_run": {"cache_hits": 4, "rows_added": 0},
  "producers": [{"lineage_id": "sha256:9c41…", "step": "current", "rows": 4, "recorded_runs": ["…"]}]
}
  • requested is the identity this run asked for: the configuration, the selected outputs, the plugin version, the image and the genome build.
  • reuse lists the lineages the run read stored scores from.
  • this_run counts the variants already scored and the rows this run wrote.
  • producers names the lineage each exported row came from, with the runs the store recorded under it.

The provenance holds hashes, versions and image digests, and no file path, bucket or credential. Score reuse and provenance describes each field.

Tell exports apart

Two runs of one model with different settings export the same column names. Compare their provenance:

  • A different requested.lineage_id for a model means its scores came from different score-changing settings.
  • The score_lineage_id column names the lineage of each row's scores.
  • A different release or genome_build in sources means a source served different data.

Read details and artifacts

By default, a model's detail tables are in a SQLite file beside its store, scores.details.sqlite for scores.sqlite, and artifacts are files under <cache_dir>/artifacts, with an index beside them. Both are in the cache, .altar, when the spec names no store. Choose where results are stored sets other places. Read both in Python. model_id is the model key, here sei, as in Select details and artifacts:

import asyncio

from altar.execution import LocalStorage
from altar.results import SqliteDetailStore, StorageArtifactStore


async def main() -> None:
    with SqliteDetailStore(".altar/scores.details.sqlite") as details:
        async for row in details.read_rows(model_id="sei", detail_name="sequence_class_effects"):
            print(row["variant_id"], row["class_label"], row["class_effect"])
    with StorageArtifactStore(LocalStorage(), ".altar/artifacts", ".altar/artifacts/index.sqlite") as artifacts:
        async for record in artifacts.read_records("sei", "chromatin_profile_effects"):
            print(len(record.variant_ids), "variants in", await artifacts.locate(record))


asyncio.run(main())

Both read every lineage the store holds for that model key unless accepted_lineage_ids limits them. Each model's page describes its detail columns and artifact layout. Store and assemble results covers the stores in full.

Write an augmented VCF

The export is a table of normalized variants, so it cannot rebuild the input VCF's records, samples and genotypes. When a run reads a VCF, you can get that VCF back with the run's values as INFO fields: pass the export and the VCF to altar.variants.export.run_export. The command below assumes a run like the quickstart's score.toml, with path = "cohort.vcf" in place of cohort.tsv:

python -m altar.variants.export.run_export \
    --export-dir first-scores/export \
    --input-vcf cohort.vcf \
    --reference-fasta hg38.fa \
    --output-vcf cohort.altar.vcf.gz

The command:

  1. reads the VCF against the FASTA, as the run did, and gives every ALT its record and its canonical variant ID;
  2. joins the export's rows onto those ALTs by variant ID;
  3. writes cohort.altar.vcf.gz, its CSI index and cohort.altar.manifest.json, and prints a summary as JSON.

The run must have read the same VCF with [variants] reference_fasta set to the same FASTA, so both left-align indels the same way. The command refuses an export whose run had no reference. The export must be Parquet. The output name must end in .vcf.gz.

The INFO fields come from the export's sources, such as ALTAR1_GNOMAD_AF and ALTAR1_CLINVAR_*, each with the release and build the export recorded, plus Altar's core fields, such as ALTAR1_STATUS and ALTAR1_PRIORITIZED. A source with no INFO preset adds no field, and the summary lists it under sources_without_preset. Write an augmented VCF lists the presets, the core fields and how to declare more fields with --annotation-fields-json.

Check the result

  • The table has as many rows as the summary's variant_count, or fewer with prioritized_only.
  • _columns.tsv has one row per column of the table.
  • Each source in the summary's sources names the release and build you expected.
  • An augmented VCF's summary counts, under occurrences, the VCF's ALTs and those that joined an export row.

If it fails

  • pd.read_parquet on a TSV export finds no Parquet file. Read part-000000000000.tsv.gz with pd.read_csv(path, sep="\t").
  • The augmented-VCF command refuses an export whose run read no reference. Add [variants] reference_fasta, rerun, and pass the same FASTA.

Fix a run that stops or warns covers messages from the run itself.

Next