Skip to content

Add a result store

A result store keeps a model's scores in a database or warehouse, and reads them back to assemble each variant's results and to export them. Altar has three store interfaces, one per result shape: ScoreStore for primary scores, DetailStore for named detail tables and ArtifactStore for artifact files. The steps below build a ScoreStore; Store named details and Store artifacts cover the other two.

When to choose a result store

Write a store when scores must live in a system the shipped stores don't reach. Altar ships sqlite and bigquery score and detail stores, and the storage artifact store, which keeps files through any Storage adapter; Platform support says how each is tested. A store holds every model's scores, so it never knows about one binding. Choose a result shape explains which store holds what.

The interface

Subclass ScoreStore, imported from altar.results. It is Stable.

Member Required? What it is
name required The registry name, and the type of a run spec's [store] section.
ensure_schema(columns) required Sets up the store to hold these score columns, adding to what it already holds.
add_scores(model_id=, model_name=, plugin_identity=, rows=, columns=) required Writes one batch of a model's rows and returns an InsertResult.
read_run_identity(model_id) required The run identity recorded for the model's scores, or None.
get_unscored_variants(variant_ids, model_id) required The requested variants the store has no score for.
read_score(model_id, variant_id) required unless materialize is overridden One variant's stored scores, or None.
read_scores(model_id, variant_ids) default: calls read_score per variant Override it with one bulk query.
primary_write(...), atomic_primary_write default: each batch written at once; False One write unit for a run's primary rows; see Write units.
materialize(...), export_download(...), export_prioritized(...) default: portable Python implementations A warehouse overrides them to run in its own query engine.
required_row_identity() default: none Locus columns the store needs on every row, such as chr and pos.
close() default: does nothing Releases what the store opened.
from_options(..., connection=) optional classmethod Builds the store from a run spec that names a connection.

The base class also has the methods for compact score lineages, such as resolve_lineages() and register_score_lineage(). A new store does not implement them; see Lineage mode.

A minimal score store

This store keeps scores in a shelve file, one entry per model and variant. It implements only the required methods and inherits materialize and the exports. Put it in a module of your own, such as lab_altar/store.py:

import math
import shelve
from typing import Any

from altar.models import IncompatibleResultError, PluginRunIdentity
from altar.results import InsertResult, SchemaCollisionError, ScoreColumn, ScoreStore


class ShelfScoreStore(ScoreStore):
    """Keep scores in a `shelve` file, one entry per model and variant."""

    name = "shelf"

    def __init__(self, path: str) -> None:
        self._shelf = shelve.open(path)
        self._open = True

    def close(self) -> None:
        if self._open:
            self._shelf.close()
            self._open = False

    async def ensure_schema(self, columns: list[ScoreColumn]) -> None:
        schema = self._shelf.get("schema", {})
        for column in columns:
            shape = (column.dtype, column.nullable)
            if schema.setdefault(column.name, shape) != shape:
                msg = f"score column {column.name!r} is already stored as {schema[column.name]}, not {shape}"
                raise SchemaCollisionError(msg)
        self._shelf["schema"] = schema

    async def add_scores(
        self,
        *,
        model_id: str,
        model_name: str,
        plugin_identity: PluginRunIdentity,
        rows: Any,
        columns: list[ScoreColumn],
    ) -> InsertResult:
        rows = list(rows)
        for row in rows:  # check every row before storing any
            for column in columns:
                value = row.get(column.name)
                if isinstance(value, float) and not math.isfinite(value):
                    raise ValueError(f"score {column.name!r} must be finite, got {value}")
        stored = await self.read_run_identity(model_id)
        current = plugin_identity.primary_result_compatibility()  # raises for an identity without scoring context
        if stored is not None and stored.primary_result_compatibility() != current:
            raise IncompatibleResultError(f"model {model_id!r} already holds scientifically incompatible scores")
        if stored is None:
            self._shelf[f"identity/{model_id}"] = plugin_identity.to_json()
        added = skipped = 0
        for row in rows:
            key = f"score/{model_id}/{row['variant_id']}"
            if key in self._shelf:
                skipped += 1
                continue
            self._shelf[key] = {column.name: row.get(column.name) for column in columns}
            added += 1
        return InsertResult(added=added, skipped=skipped)

    async def read_run_identity(self, model_id: str) -> PluginRunIdentity | None:
        stored = self._shelf.get(f"identity/{model_id}")
        return None if stored is None else PluginRunIdentity.from_json(stored)

    async def get_unscored_variants(self, variant_ids: Any, model_id: str) -> list[str]:
        return [variant_id for variant_id in variant_ids if f"score/{model_id}/{variant_id}" not in self._shelf]

    async def read_score(self, model_id: str, variant_id: str) -> dict[str, Any] | None:
        return self._shelf.get(f"score/{model_id}/{variant_id}")

What each part does:

  • ensure_schema records each column's type and nullability, and refuses a column already stored with another.
  • add_scores checks every value before it stores any row, and refuses NaN and infinity with ValueError. It calls primary_result_compatibility(), which raises IncompatibleResultError for an identity without a model instance's scoring context. It refuses rows for a model ID whose recorded identity is scientifically different, and records the first identity. It skips a row it already holds and counts it in skipped.
  • read_run_identity, get_unscored_variants and read_score read back what add_scores wrote. read_score returns only the model's own columns.

Schema rules

Store the variant ID, the model ID and the complete run identity apart from the declared score fields. Refuse a second, incompatible identity for an existing model ID. Refuse rows whose identity has no scoring context, rather than giving them the current binding's meaning.

A score is a finite value, or None when it is missing. Reuse a score column name across models only when its type and nullability agree with the stored column; labels may differ. Refuse an incompatible column before writing any data.

If the store needs locus columns on every row, return their types from required_row_identity(). The scoring engine derives chr (str), pos (int), ref and alt (str) from the canonical variant_id when a row lacks them, and checks them when it has them, so model bindings never learn which store needs them. Don't redefine those types. Any other required column must come from the binding's result codec, scoring_result_codec().

Document whether a repeated write is skipped or appended. The SQLite store skips a row it already holds, and the BigQuery store appends it. Callers find the variants to score with get_unscored_variants() before writing, and don't rely on add_scores to skip duplicates.

Write units

The scoring engine writes a run's primary rows through one write unit, after the run's details and artifacts, so a variant's primary row marks it as complete:

async with store.primary_write(
    model_id=model_id, model_name=model_name, plugin_identity=identity, columns=columns
) as writer:
    await writer.add(rows)

The default unit calls add_scores for each batch, so each batch is stored as soon as add returns. An exception inside the block propagates, and the batches already added stay. A store that can publish a whole unit at once, such as in one database transaction or by promoting a staging table, overrides primary_write to yield its own PrimaryScoreWriter and sets atomic_primary_write = True. A clean exit then commits every batch, and an exception commits none. The SQLite and BigQuery stores do this. A store that subclasses another and changes how add_scores writes must override primary_write too.

Lineage mode

A store runs in one of two modes:

  • The original mode keeps one compatible run identity per model ID. It answers cache lookups with read_run_identity and get_unscored_variants. It refuses a ScoreReusePolicy that accepts other lineages, with CapabilityNotSupported.
  • Compact-lineage mode keys every row by its score lineage, keeps a catalog of lineages and the exact identities that wrote under each, and lets a caller accept earlier lineages. SqliteScoreStore(path, lineages=True) and BigQueryScoreStore(..., lineage_table_id=...) turn it on.

The storage interface behind compact-lineage mode is not public, so a new store runs in the original mode. Its lineage methods raise CapabilityNotSupported, and the conformance suite skips the lineage tests unless you override its lineage_store fixture. Reuse scores across model updates describes what compact lineages give a caller.

Close what the store opens

ScoreStore, DetailStore and ArtifactStore each have a close() method, and a with block calls it on exit. The default does nothing. A store that opens a connection or creates a client overrides it to release them, and a second call must do nothing. If the constructor fails after opening a connection, close the connection before raising.

A store closes only what it opened. A client or storage adapter passed to the constructor belongs to the caller, who closes it. A store whose from_options builds a client from a connection owns that client, as BigQueryScoreStore does.

altar run closes each store it builds from the spec once, when the run ends: after the export, after a failed check or model, and when the run is cancelled.

Assemble results in the warehouse

A warehouse may override materialize() to run the joins and the prioritization in its own query engine, and export_download() and export_prioritized() to export in bulk. The result must match the portable implementation: one row per distinct requested variant in variant_id order, the same missing values, the same model and source attribution, and three-valued predicate logic. The conformance suite checks each of these. BigQueryScoreStore is the reference.

Produce an export's result buckets

Every export writer reads a store through result_buckets(), so a store that implements read_score exports in every layout with nothing more: the default reads each bucket of variants through materialize. A warehouse may override result_buckets() to produce the buckets from its own query, as BigQueryScoreStore does. It is an async context manager that yields a ResultBuckets, and it removes on exit whatever it created for the export, such as a temporary table.

A bucket is an Arrow table with result_bucket_schema(layout), schema version RESULT_BUCKET_SCHEMA_VERSION, one row per variant:

Column Holds
variant_ordinal The variant's 0-based place in genomic order (genomic_sort_key), the sort key
variant_id, prioritized The canonical ID and whether a model or source kept the variant
rider columns A display rider's columns, when one is given, computed by the store
annotation columns Each declared column once per variant, in its type; no value is null
model_scores One entry per model that scored the variant, in the order of models: model_id, model_name, score_lineage_id, the model's prioritized, and every model's score fields, null where the model has no such field
prioritized_model_ids, prioritized_source_names Why the variant was kept

A ResultBucketDescriptor gives each bucket's first and last ordinal and its row count. Buckets come in ordinal order; the rows in a bucket may come in any order. Cut buckets with plan_buckets over the sizes bucket_rows_for returns, from the layout alone, so every store draws the same boundaries and writes the same bytes. A rider's pick reads a missing decision as not prioritized and breaks a tie by model order. The conformance suite renders every layout from your buckets and checks them byte for byte against a reference store, on a local and an in-memory storage. These names are provisional.

Register the entry point

Register a score store under altar.score_stores, with its name as the key:

[project.entry-points."altar.score_stores"]
shelf = "lab_altar.store:ShelfScoreStore"

A run spec then selects it with type = "shelf" and options = { path = "scores" } in its [store] section.

Build it from a run spec

The run driver builds the store a spec's [store] section names in one of two ways:

  • Without a connection, it passes the section's options to the constructor as keywords. A store whose constructor needs a client cannot be built this way.
  • With connection = "<name>", it calls the store's classmethod from_options(**options, connection=...). The store builds its client from the connection, as BigQueryScoreStore.from_options does with connection.bigquery_client(). Only a store whose from_options names a connection parameter takes one; **options alone does not count. A connection never enters a score's identity.

Add connections and published datasets covers connections, which are provisional.

Test with the conformance suite

Subclass ScoreStoreContract from altar.testing and give it a store fixture that returns a fresh, empty store. A store that holds a connection is yielded and closed after the test:

import pytest
from altar.testing import ScoreStoreContract

from lab_altar.store import ShelfScoreStore


class TestShelfScoreStore(ScoreStoreContract):
    @pytest.fixture
    def store(self, tmp_path):
        with ShelfScoreStore(str(tmp_path / "scores")) as store:
            yield store

The suite checks schema setup, writes and their counts, identity conflicts, refused non-finite scores, duplicate writes, write units, closing twice, that materialize and both exports give the portable results, with annotation sources joined, and that the result buckets render every export layout to the reference store's bytes. A store with a compact-lineage mode also overrides the lineage_store fixture. Override make_annotation_source when the store can join only sources that live on its own system. Add your own tests for schema evolution and for several model architectures in one store.

Store named details

Implement DetailStore when the system should hold the repeated rows a binding declares as detail tables. It is Stable. Its required methods create or check a named table (ensure_table), find missing row keys (get_missing_keys), add rows (add_rows), stream rows by model, table and variant (read_rows), and report the stored schema and plugin identity of a table (describe_table). Every read takes the lineages whose rows count. get_missing_variants has a default that streams read_rows; override it with an indexed query. Don't add model-specific methods or pair a store with one binding.

The SQLite store keeps canonical JSON rows behind indexed model, table, variant and row-key columns. The BigQuery store uses the same shape in fixed tables, with staged MERGE writes. A store may use native columns instead only if it still detects schema and plugin collisions exactly, and can add any binding's schema without an ambiguous migration. Register it under altar.detail_stores, and test it with DetailStoreContract. A run spec selects it in [detail_store], built the same way as the score store.

Store artifacts

Implement ArtifactStore when the system should hold artifact files. It is Stable. It stores each file without decoding it, keyed by the model, the artifact's name and the producing run's score lineage, with the variants the file covers. Its required methods are ensure_artifact, put, get_missing_variants, read_records, locate and describe_artifact. StorageArtifactStore is the reference: it writes files through a Storage adapter and keeps its index in SQLite.

Register it under altar.artifact_stores, and test it with ArtifactStoreContract. A run spec selects it in [artifact_store]. The run driver passes the run's storage, and a root_uri and index_path in the cache, [run] cache_dir, to a constructor that takes them and isn't given them.

Ready when

  • The store refuses an incompatible identity or column for an existing model before writing any row.
  • add_scores refuses NaN and infinity, and its idempotency is documented.
  • read_score or read_scores, or a materialize override, returns each model's own columns only.
  • A write unit is atomic, with atomic_primary_write = True, or the store keeps the default.
  • close() releases only what the store opened, and a second call does nothing.
  • The entry point's key equals name.
  • ScoreStoreContract passes, and DetailStoreContract or ArtifactStoreContract for those stores.
  • The page that documents the store says what it was tested against.

Next