Skip to content

Validate an implementation

altar.testing publishes a conformance suite for each extension interface: a pytest class you subclass and give a few fixtures. Altar's own backends, stores and bindings run the same suites. The store suites score with ContractModelPlugin, a synthetic model with scalar and detail results, so a store is tested against Altar's storage rules alone. A suite proves an implementation behaves as Altar expects. It does not prove its science; add your own tests for that (see Add tests of your own).

The suites need pip install pytest pytest-asyncio, with asyncio_mode = "auto" under [tool.pytest.ini_options].

Available contracts

Contract Checks an implementation of Shown in
PluginDiscoveryContract any plugin's installed entry point Add a model binding
ModelResultContract (also exported as ModelPluginContract, a stable alias) a ModelPlugin's manifest and results Add a model binding
InlineScorerContract a ModelPlugin that scores in process, including with_run_variants Add an inline model binding
ContainerScorerContract a ModelPlugin's container scoring plan Add a model binding
ExecutionBackendContract ExecutionBackend Add an execution backend
ScoreStoreContract ScoreStore Add a result store
DetailStoreContract DetailStore Store named details
ArtifactStoreContract ArtifactStore Store artifacts
AnnotationSourceContract AnnotationSource Add an annotation source
SqlAnnotationSourceContract SqlAnnotationSource Join a source in BigQuery
WritableAnnotationSourceContract an annotation source that is written to (opt-in) A source Altar writes to
AllelicRecordBackendContract AllelicRecordBackend Fixtures
VariantGeneLinkSourceContract VariantGeneLinkSource Variant-to-gene links
VariantGeneLinkAnnotationSourceContract a source of the variant–gene link annotation contract Fixtures
VariantGeneLinkStoreContract VariantGeneLinkStore Variant-to-gene links
StorageContract Storage Fixtures
ConnectionContract (provisional) Connection A connection type
PublishedDatasetContract (provisional) an annotation source whose publisher hosts its data A publishing binding
JobQueueContract (provisional) JobQueue
AuthProviderContract (provisional) AuthProvider
InterpretableModelContract (provisional) an InterpretableModel's interpretation methods and workflow Add interpretation to a model binding
InterpretationStoreContract (provisional) InterpretationStore

The six provisional suites test provisional contracts, and are expected to change with them.

Fixtures of other suites

AllelicRecordBackendContract takes a backend, a query that projects at least two fields, and the expected_record_count it returns. ArtifactStoreContract takes a fresh store; it writes its source files to tmp_path. VariantGeneLinkAnnotationSourceContract is AnnotationSourceContract plus the link projection rules, so it takes the same source and known_variant_ids fixtures. assert_link_projection checks one row the same way, for rows your own code loads elsewhere.

StorageContract takes a storage and a writable base_path. Two more fixtures are optional, and the tests that need them are skipped when they are not overridden:

  • list_names returns a function that lists the file names directly inside a storage directory. With it, the suite checks that writes leave no temporary files behind.
  • interrupt_writes returns a function that takes a byte count and an exception and returns a context manager. Inside the context, each write or upload stores that many bytes of its new content, then raises the exception. With it, the suite checks that an interrupted overwrite keeps the previous file and that an interrupted first write creates no file. Supply it only if your adapter guarantees both.

LocalStorage supplies both fixtures. FsspecStorage supplies only list_names, because fsspec does not guarantee that a failed write keeps the previous file.

Test doubles

altar.testing also has in-memory stand-ins for the interfaces a binding's tests touch:

Double Stands in for
InProcessExecutionBackend (provisional) an ExecutionBackend: it runs each task by calling a Python function instead of a container
InMemoryScoreStore a ScoreStore
InMemoryDetailStore (provisional) a DetailStore
InMemoryAnnotationSource an AnnotationSource
InMemorySqlAnnotationSource a SqlAnnotationSource
InMemoryAllelicRecordBackend an AllelicRecordBackend
InMemoryVariantGeneLinkSource a VariantGeneLinkSource
InMemoryJobQueue (provisional) a JobQueue
FakeBigQueryClient (provisional) a BigQuery client, recording what it was asked to run

InProcessExecutionBackend lets a container binding test run_scoring() end to end without Docker. The function it calls stands in for the runtime: it reads the staged inputs and writes the runtime's result files. Add a model binding shows it.

Use a contract

import pytest
from altar.testing import AnnotationSourceContract


class TestMySource(AnnotationSourceContract):
    @pytest.fixture
    def source(self):
        return MySource(FakeBackend())

Read the selected contract's fixture requirements and supply only those fixtures. An annotation source that implements a published AnnotationContract also returns it from the contract fixture, and the suite checks the source against it with assert_implements_contract. An annotation source that is also written to, such as a cache of annotations your own pipeline computes, additionally subclasses WritableAnnotationSourceContract. It checks that writes are stored under the source's identity, that a source with another identity sees none of them as annotated, and that reads never mix identities. Keep these tests in the extension package, so each Altar version you support is checked against them.

ContractModelPlugin and contract_model_identity() are available when a store or backend test needs the same neutral schema or a complete deterministic run identity. Model bindings test their real plugin instead.

A model binding runs PluginDiscoveryContract, ModelResultContract, and either InlineScorerContract or ContainerScorerContract, plus one test that scores through run_scoring(). An InterpretableModel also runs InterpretableModelContract. Annotation sources, variant–gene link sources, execution backends and stores use the suite of their own interface.

Suites the shipped bindings run

Binding Suites
altar-alphagenome discovery, results, inline scorer
altar-borzoi, altar-enformer, altar-legnet, altar-sei discovery, results, container scorer
altar-cherimoya, altar-chrombpnet discovery, results, container scorer, interpretable model
altar-alphamissense, altar-gpnstar, altar-spliceai discovery, annotation source
altar-clinvar, altar-popfreq discovery, annotation source, SQL annotation source, published dataset
altar-conservation discovery, annotation source, published dataset
altar-e2g-atlas discovery, variant–gene link source
altar-opentargets-e2g discovery, variant–gene link source, variant–gene link annotation source, published dataset

Add tests of your own

Conformance proves shared behavior, not scientific correctness. Also test:

  • exact schemas, units, and field direction;
  • supported and rejected variants;
  • missing, zero, null, and duplicate records;
  • provenance and genome-build mismatches;
  • deterministic task commands and resource requests;
  • importability without optional SDKs, credentials, GPUs, or weights;
  • entry-point discovery in an installed environment.

Type-check a binding against Altar's public modules only, and test it in a clean environment that holds only the package and its declared dependencies.

Next