Skip to content

Score a cohort

altar.scoring scores variants with a model you configure in code and writes the rows to stores you hold. altar run makes the same calls. Use them directly when your own application keeps its stores, its execution backend or its job records.

Call What it does
score_cohort Prepares a cohort's cache misses, then runs every batch.
prepare_scoring_batches Filters a cohort against the stores and packs the misses into requests.
run_scoring Runs one request and stores its rows.
executable_scoring_plan Returns the plan for one request, for code that submits the plan's tasks itself.
ingest_scoring_results Stores the outputs of a plan you ran on your own backend.

A binding turns each request into a plan: the container tasks, or the function in your process, that compute the scores. Container plans and Inline plans describe the two kinds.

These names are stable, except ScoringOutputError, which is provisional. Signatures are in the scoring engine reference.

Build a request

A ScoringRequest names one model instance and the analysis it runs in:

Field What it holds
job_id, model_id The job and the model instance; altar run uses the model key, the <name> in [models.<name>], as model_id.
configuration The binding's configuration class: weights, settings, runtime image. It is part of the scores' identity.
run_inputs The binding's run-input class: the genome build, the reference genome, and for an inline model the variants.
output An OutputSelection: the reducer and the outputs to write. plugin.default_output is the binding's default.
resolver A PathResolver that places logical paths under a backend's mount root: backend.path_resolver().

Each model page names its binding's configuration and run-input classes. To build them from plain values, as a spec does, call plugin.parse_configuration(mapping) and plugin.parse_run_inputs(mapping). altar.models.get_model_plugin("LEGNET") resolves an installed binding by the type a spec names.

This example scores one variant with the AlphaGenome binding. It replaces the hosted API call with a local function, so it runs offline and needs no API key. Install altar-alphagenome first.

import asyncio

from altar.execution import PathResolver
from altar.models import ScoringRequest
from altar.results import SqliteDetailStore, SqliteScoreStore
from altar.scoring import run_scoring
from altar_alphagenome import AlphaGenomeConfiguration, AlphaGenomePlugin, AlphaGenomeRunInputs, AlphaGenomeRuntime


def fake_api(variant):
    """Stand in for the hosted API: one liver RNA-seq track for every variant."""
    return [
        {
            "output_type": "RNA_SEQ",
            "track_name": "liver",
            "ontology_curie": "UBERON:0002107",
            "raw_score": -0.8,
            "quantile_score": 0.95,
        }
    ]


async def main() -> None:
    plugin = AlphaGenomePlugin()
    request = ScoringRequest(
        job_id="demo",
        model_id="alphagenome-liver",
        configuration=AlphaGenomeConfiguration(ontology_terms=("UBERON:0002107",)),
        run_inputs=AlphaGenomeRunInputs(
            genome_build="hg38",
            variants=[{"variant_id": "chr7:140753336:A:T", "chr": "chr7", "pos": 140753336, "ref": "A", "alt": "T"}],
        ),
        output=plugin.default_output,
        resolver=PathResolver(),
    )
    with SqliteScoreStore("scores.sqlite", lineages=True) as scores, SqliteDetailStore("details.sqlite") as details:
        result = await run_scoring(
            plugin,
            request,
            model_name="AlphaGenome liver",
            score_store=scores,
            detail_store=details,
            runtime=AlphaGenomeRuntime(score_variant_fn=fake_api),
        )
    print(result.added, result.skipped)


asyncio.run(main())

The first run prints 1 0: one row added. A second run prints 0 1, because the store already holds that row. AlphaGenome writes a per-track detail table, so the call needs a DetailStore as well as the ScoreStore. run_scoring refuses to start when a store the binding needs is missing.

run_scoring computes every variant in the request, even one the store already holds, and then skips writing it. To score a cohort, prepare it first.

Prepare cache misses before requests

prepare_scoring_batches reads the whole cohort, finds the variants the stores do not hold, and packs only those into requests. Filtering each batch on its own leaves undersized runs: 399 misses scattered through 1,000 candidates become batches of 100, 100, 100 and 99 when the limit is 100. Preparation:

  1. reads the candidate file and drops repeated variants;
  2. sets aside the variants the model cannot score (see Variant eligibility);
  3. derives the run's identity and refuses a store that holds incompatible results, before staging anything;
  4. queries the stores in chunks of cache_query_size variants;
  5. numbers the misses into batches of max_variants_per_batch, in the cohort's order;
  6. for a container model, stages one variant file per batch; for an inline model, binds each batch into the request's run inputs;
  7. builds one request per batch, once the total miss count is known.

The candidate file is headerless, tab-separated UTF-8 with the columns chr, pos, ref, alt and an optional fifth ID column. It is read with the same reader every model runtime uses: blank lines are skipped, aliases such as 1 and chr01 become chr1, and an invalid row fails with its line number. The cohort and the miss set live in a temporary SQLite file, not in memory.

This example prepares a three-row cohort for the AlphaGenome model above and runs each batch. fake_api is the function from the first example. The third row repeats the first in another spelling:

import asyncio
from dataclasses import replace
from pathlib import Path

from altar.execution import LocalStorage, PathResolver, Transfer
from altar.models import ScoringRequest
from altar.results import SqliteDetailStore, SqliteScoreStore
from altar.scoring import prepare_scoring_batches, run_scoring
from altar_alphagenome import AlphaGenomeConfiguration, AlphaGenomePlugin, AlphaGenomeRunInputs, AlphaGenomeRuntime


def fake_api(variant):
    return [
        {
            "output_type": "RNA_SEQ",
            "track_name": "liver",
            "ontology_curie": "UBERON:0002107",
            "raw_score": -0.8,
            "quantile_score": 0.95,
        }
    ]


def staged(transfer: Transfer) -> Transfer:
    """Place each file preparation writes under prepared/."""
    return replace(transfer, uri=str(Path("prepared", transfer.logical_path).resolve()))


async def main() -> None:
    Path("cohort.tsv").write_text("chr7\t140753336\tA\tT\nchr1\t1000000\tA\tG\n7\t140753336\ta\tt\n")
    plugin = AlphaGenomePlugin()
    template = ScoringRequest(
        job_id="demo",
        model_id="alphagenome-liver",
        configuration=AlphaGenomeConfiguration(ontology_terms=("UBERON:0002107",)),
        run_inputs=AlphaGenomeRunInputs(genome_build="hg38"),
        output=plugin.default_output,
        resolver=PathResolver(),
    )
    with SqliteScoreStore("scores.sqlite", lineages=True) as scores, SqliteDetailStore("details.sqlite") as details:
        preparation = await prepare_scoring_batches(
            plugin,
            template,
            candidate_variants=Transfer(uri="cohort.tsv", logical_path="candidates.tsv"),
            score_store=scores,
            detail_store=details,
            storage=LocalStorage(),
            destination_transfer=staged,
            max_variants_per_batch=100,
        )
        print(preparation.candidate_count, preparation.cache_hit_count, preparation.unscored_count)
        for batch in preparation.batches:
            result = await run_scoring(
                plugin,
                batch.request,
                model_name="AlphaGenome liver",
                score_store=scores,
                detail_store=details,
                runtime=AlphaGenomeRuntime(score_variant_fn=fake_api),
            )
            print(batch.request.scoring_batch, result.added)


asyncio.run(main())

On a fresh store it prints 2 0 2 and then 0 2: two unique candidates, no cache hits, one batch that adds two rows. Run it again and it prints 2 2 0, with no batch to run. preparation.duplicate_count counts the repeated row.

The template request must not name a batch; preparation numbers the batches. destination_transfer must place each file at a storage location: a mapper that leaves the URI equal to the logical path is refused before anything is staged. max_variants_per_batch sets the size of one request. A model runtime may still split a request into smaller inference batches of its own.

score_cohort makes the same preparation and then runs every batch. It takes the arguments of both calls and returns a CohortScoringResult with the preparation, each batch's ScoringRunResult in runs, and the totals added, skipped and variants_without_rows. With the imports, fake_api and staged of the example above, this function replaces its preparation and loop:

from altar.scoring import score_cohort


async def score(plugin, template, scores, details):
    cohort = await score_cohort(
        plugin,
        template,
        candidate_variants=Transfer(uri="cohort.tsv", logical_path="candidates.tsv"),
        storage=LocalStorage(),
        score_store=scores,
        detail_store=details,
        model_name="AlphaGenome liver",
        max_variants_per_batch=100,
        destination_transfer=staged,
        runtime=AlphaGenomeRuntime(score_variant_fn=fake_api),
    )
    print(cohort.added, cohort.preparation.cache_hit_count)
    return cohort

A batch that fails stops the cohort. The batches before it are stored, so calling score_cohort again with the same cohort finds them as cache hits and resumes at the failed batch. For a container model, score_cohort maps each batch's staged variant file into the plan for you; with run_scoring, map it in input_transfer yourself (see Container plans).

reuse_policy= accepts older scores of the same model; see Reuse scores across model updates. A store created with lineages=True keeps each computation's scores apart as a lineage. With such a store, preparation.resolution lists the lineages every cache check read. Read the job's results back with ScoreReusePolicy.exactly(preparation.resolution.lineage_ids), as altar run does for its export.

A warehouse can prepare the same misses in SQL without downloading the cohort. See Run at warehouse scale on BigQuery.

Variant eligibility

A model's manifest may declare the variants it can score, its variant_eligibility. Enformer, Borzoi, LegNet and Sei score SNVs only; the other bundled models admit every variant. Preparation sets the other candidates aside before it queries the cache, so they never reach a runtime.

After the manifest's rule, preparation checks each admitted candidate against the reference and against allele normalization, and sets it aside with the first reason that applies:

Reason When
unsupported_variant_class The manifest does not admit the variant's class, such as an indel for an SNV-only model.
allele_too_long An allele is longer than the manifest allows.
contig_missing reference_fasta has no contig for the variant's chromosome.
reference_mismatch REF is not the reference bases at the variant's position.
not_normalized The variant is not in its normalized form: left-aligned with reference_fasta, trimmed without it.

The two reference checks need reference_fasta. Preparation never rewrites an ID. altar run normalizes the cohort as it reads it, so its candidates pass the normalization check; see Variant identity and normalization.

Preparation reports the set-aside variants three ways:

  • preparation.ineligible_count, so cache_hit_count + unscored_count + ineligible_count equals candidate_count;
  • preparation.ineligible_variants, a report staged through destination_transfer at ScoringLayout.ineligible_variants_file(job_id, model_id), or None when every candidate is eligible;
  • the report's columns, INELIGIBLE_VARIANT_COLUMNS: chr, pos, ref, alt, variant_id, variant_class, reason, in the cohort's order.

Altar stores nothing for an ineligible variant, so it is not a cache miss and never causes rescoring. Each preparation sets the same variants aside again. In assembled results, an ineligible variant looks like an unscored one: no score from that model.

The engine applies the same rule to output. When a runtime returns a primary or detail row for an ineligible variant, run_scoring raises ResultValidationError. An inline result is checked whole before any table is written. A container result is checked one batch at a time, before that batch is written, so batches written before it stay stored. An inline binding that restricts eligibility filters its own inputs with manifest.variant_eligibility.admits().

Bindings with named details

A binding with named detail tables, such as Borzoi's gene_effects, Enformer's track_effects or AlphaGenome's track_scores, needs the DetailStore that scoring writes to. Preparation refuses to start without it. A variant is a cache hit only when:

  1. the score store holds its primary row under an accepted lineage; and
  2. every declared detail table holds at least one row for it under an accepted lineage, in a table with the same name, version and field meanings. A stored table of the current lineage with another meaning raises IncompatibleResultError before any batch is staged.

Every other variant is a miss and is scored again. The engine skips the primary and detail rows that already exist, so repairing a partial result never duplicates a row. preparation.incomplete_detail_count counts the misses whose primary row was cached but whose detail rows were not.

The rule holds because run_scoring writes every detail table before any primary row of the same run. A failure while writing details leaves no primary row, so the primary row marks a variant complete. Two consequences:

  • A variant that produces no rows for a table, such as a Borzoi variant with no gene in its window, is scored again on every preparation, unless the table's schema names a row_count_field and the cached primary row holds 0 in it. The extra run writes nothing new.
  • A variant with some detail rows and a primary row counts as complete. Code that writes primary rows another way must write the complete detail rows first.

A request whose output selection names artifacts needs the artifact_store scoring writes to, and a variant is then a hit only when every selected artifact covers it as well.

Container plans

A container binding returns a ContainerScoringPlan: independent ContainerTaskSpec tasks, often one per fold, plus an optional dependent task that summarizes them. Each task declares its image and command, its CPU, memory and GPU request, its labels, and the logical files it reads and writes. An execution backend runs the tasks.

run_scoring needs the backend and a storage for a container plan. Two mappers place the plan's logical files in your deployment:

  • input_transfer maps each input the plan declares to the URI it is read from: the batch's variant file and any resource you stage yourself.
  • output_transfer maps each output to the URI it is collected to.

This function runs every batch of a container model's preparation on local Docker:

from dataclasses import replace

from altar.execution import LocalDockerExecutionBackend, LocalStorage
from altar.scoring import run_scoring


def collected(transfer):
    """Collect every output under a results directory."""
    return replace(transfer, uri=f"/data/results/{transfer.logical_path}")


async def run_batches(plugin, preparation, *, backend: LocalDockerExecutionBackend, scores):
    for batch in preparation.batches:

        def staged(transfer, prepared=batch.variant_input):
            """Read the batch's variant file where preparation staged it; leave model resources as declared."""
            return prepared if transfer.logical_path == prepared.logical_path else transfer

        await run_scoring(
            plugin,
            batch.request,
            model_name="LegNet WTC11",
            score_store=scores,
            backend=backend,
            storage=LocalStorage(),
            input_transfer=staged,
            output_transfer=collected,
        )

Build the backend with LocalDockerExecutionBackend(data_root=...), and the template request with resolver=backend.path_resolver() so its paths fall under the backend's mount root. Before it submits anything, run_scoring deletes any file already at an output location, so it never reads an earlier attempt's file. A terminal output still missing after collection raises ScoringOutputError.

spec_mapper= adjusts each task before submission, for example to raise a timeout or a memory request. It may change only resources, timeout_s and avoid_hosts. The image, command, transfers, labels and identity stay as the binding wrote them. Pass a TaskLedger as ledger= to record each submitted task; see Track task state.

run_scoring checks the batch's tasks every poll_interval_s seconds (5 by default) and waits at most timeout_s seconds (24 hours by default). Tasks still open then are cancelled through the backend, and it raises TimeoutError; nothing of the batch is stored. A backend whose cancel does nothing, such as the Modal backend, leaves them running until their own limit. Pass progress= a function that takes one string to see the wait: it receives a line when the tasks are submitted, naming the time limit, then the counts of running, waiting and finished tasks when they change, at most every 15 seconds, and at least once a minute. score_cohort takes the same three arguments, starts with a line saying how many variants it scores in how many batches, and puts batch <n> of <total>: before each batch's lines. The time limit applies to each batch, not to the whole cohort:

await score_cohort(..., timeout_s=6 * 3600, progress=lambda line: print(f"[legnet] {line}"))
[legnet] 12,000 variants to score, in 3 batches
[legnet] batch 1 of 3: 4 tasks submitted; waiting up to 6h
[legnet] batch 1 of 3: 4 of 4 tasks running, 15s elapsed

Inline plans

An inline binding returns an InlineScoringPlan: an async scorer that runs in the calling process, such as a call to a hosted API. It needs no backend, storage or transfer mappers. Pass the binding's runtime context as runtime=; it carries what a spec cannot, such as an API client or a secret resolver. The examples above use AlphaGenome's.

An inline binding receives its variants in its run inputs. Preparation binds each batch into the request with the binding's with_run_variants, so prepare_scoring_batches and score_cohort work for inline models as they do for container models.

Check and write the rows

For a container plan, the binding's decoder, scoring_result_codec(), reads the runtime's result files in bounded batches. Each row holds variant_id and the manifest's score fields. The default decoder reads a headered TSV whose columns are the score names. A binding overrides it when its runtime writes other names: ChromBPNet maps its runtime's lfc to logfc. When a store needs chr, pos, ref and alt columns (required_row_identity()), the engine derives any the runtime left out from variant_id and rejects any that disagree with it.

Before it writes, the engine rejects:

  • a missing or undeclared field, or a missing column the store requires;
  • a value of the wrong type, or a float that is not finite;
  • a repeated variant_id;
  • a row for a variant the batch did not ask for (ScoringOutputError);
  • a result schema that differs from the manifest.

It writes in this order: the selected artifacts, then every detail table, then the primary rows in one write unit. On a store whose write unit is atomic, such as BigQueryScoreStore, a failed run commits no primary row. Each row carries the run's PluginRunIdentity. A store refuses a model_id that already belongs to another model instance: another configuration, output selection, genome build, runtime or result schema. Job IDs, paths and batch boundaries are not part of the identity, so separate batches and repeated runs fill one cache.

Before each write the engine asks the store which rows it lacks and skips the others. That check is not a transaction across processes. Two processes scoring the same model into one store can both write a row; run one writer per model at a time, or deduplicate afterwards. result.variants_without_rows counts the batch's variants that got no primary row.

The fields plugin.unavailable_score_fields names for the request are stored as null. A model trained on another assembly than the variants' uses this for fields that do not carry over.

Detail tables follow the same path. Each DetailSchema declares a table name, a version, its fields and its key fields. The engine checks every row, rejects a key repeated across batches, and writes the rows under the run's lineage. An artifact is a file the engine stores without reading it, through the ArtifactStore. See Store and assemble results for reading them back.

Ingest outputs you collected

ingest_scoring_results is the ingest step of run_scoring without the execution. When your own code submits and tracks a container plan's tasks, call it once the outputs are collected. It decodes, checks and writes them as run_scoring does. An inline plan has no outputs to collect, so the call raises TypeError for one; run it with run_scoring.

executable_scoring_plan(plugin, request, storage=..., input_transfer=...) returns the plan to submit. It is the plan run_scoring runs. When the request selects a per-variant artifact, it reads the batch's variant file to list one output per variant, so pass the storage and input_transfer that find that file.

Before you submit a task, delete any file at its output locations, so the ingest never reads an earlier attempt's file.

Freeze the plan you submit

Without plan=, the ingest rebuilds the plan from the request. The rebuilt plan matches the submitted one only while the installed binding stays the same. A binding upgrade can change a default image or setting, and the rebuilt plan then describes another run.

When work can outlive a binding upgrade, freeze the plan when you accept the work, and pass that plan to the ingest:

from altar.models import scoring_plan_from_json, scoring_plan_to_json
from altar.scoring import executable_scoring_plan, ingest_scoring_results


async def admit(plugin, request, *, storage, staged):
    """Build the plan once and keep its JSON; then submit the plan's tasks."""
    plan = await executable_scoring_plan(plugin, request, storage=storage, input_transfer=staged)
    return scoring_plan_to_json(plan)


async def complete(snapshot, plugin, request, *, scores, storage, staged, collected):
    """Store the collected outputs, in any process and after any binding upgrade."""
    return await ingest_scoring_results(
        plugin,
        request,
        model_name="K562 DNase",
        score_store=scores,
        storage=storage,
        input_transfer=staged,
        output_transfer=collected,
        plan=scoring_plan_from_json(snapshot),
    )

The ingest stores the rows under the frozen plan's identity. Before it writes anything, it checks the plan against the installed binding and the request:

  • The plugin ID, kind, version and result schema must be the installed binding's, and the plan must support the running Altar API.
  • The request must describe the same run. Its configuration, output selection, genome build and genome, with the plan's runtimes, must give the plan's lineage. A binding that names a runtime image in its configuration must name the plan's image.
  • A task label for the job, model, scoring batch or batch count must match the request.
  • The plan's result files must cover the primary result, every detail table and each artifact the request selects.
  • The plan must stage the batch's variant file, and storage must find it where input_transfer maps it.

The exact configuration_hash may differ from the request's, for example after the binding moved saved configurations to a newer schema version. The rows keep the plan's. Any other mismatch raises before a store is touched:

Mismatch Raises
The binding or the run IncompatibleResultError
Another job or batch ValueError
Result files ResultValidationError
A missing variant file ScoringOutputError

Build the request at completion from the saved configuration the plan was built from. A configuration that picks up a binding's new default image no longer describes the frozen run, and the ingest refuses it. Ingesting the same outputs again skips every row and file already stored.

Collect and ingest a frozen batch

Code that decodes runtime files and writes the rows itself must repeat every check above. Hand that step to the engine instead. This completion step collects a batch's outputs, ingests them under the frozen plan, and refuses an empty result:

from altar.models import scoring_plan_from_json
from altar.scoring import ingest_scoring_results


async def ingest_frozen_scores(snapshot, *, plugin, model_name, backend, storage, store, staged, collected):
    # snapshot: your record of one batch, holding the plan's JSON and what the request was built from.
    # staged: maps a plan input, such as the batch's variant file, to where you staged it.
    # collected: maps a terminal output to where your backend collects it.
    plan = scoring_plan_from_json(snapshot.plan_json)
    request = snapshot.scoring_request(plugin)  # rebuilt from the saved configuration the plan was built from
    terminal = [plan.summarize] if plan.summarize is not None else plan.shards
    await backend.collect_outputs([collected(output) for task in terminal for output in task.outputs])
    result = await ingest_scoring_results(
        plugin,
        request,
        model_name=model_name,
        score_store=store,
        storage=storage,
        input_transfer=staged,
        output_transfer=collected,
        batch_size=250_000,
        plan=plan,
    )
    if result.added + result.skipped == 0:
        msg = "the scorer returned no rows; refusing to mark scoring complete"
        raise ValueError(msg)
    return result

To refuse a partial result as well, check result.variants_without_rows, the batch's variants that got no primary row.

Next