Validate an implementation¶
altar.testing publishes a conformance suite for each extension interface: a pytest class you subclass and give
a few fixtures. Altar's own backends, stores and bindings run the same suites. The store suites score with
ContractModelPlugin, a synthetic model with scalar and detail results, so a store is tested against Altar's
storage rules alone. A suite proves an implementation behaves as Altar expects. It does not prove its science; add
your own tests for that (see Add tests of your own).
The suites need pip install pytest pytest-asyncio, with asyncio_mode = "auto" under [tool.pytest.ini_options].
Available contracts¶
| Contract | Checks an implementation of | Shown in |
|---|---|---|
PluginDiscoveryContract |
any plugin's installed entry point | Add a model binding |
ModelResultContract (also exported as ModelPluginContract, a stable alias) |
a ModelPlugin's manifest and results |
Add a model binding |
InlineScorerContract |
a ModelPlugin that scores in process, including with_run_variants |
Add an inline model binding |
ContainerScorerContract |
a ModelPlugin's container scoring plan |
Add a model binding |
ExecutionBackendContract |
ExecutionBackend |
Add an execution backend |
ScoreStoreContract |
ScoreStore |
Add a result store |
DetailStoreContract |
DetailStore |
Store named details |
ArtifactStoreContract |
ArtifactStore |
Store artifacts |
AnnotationSourceContract |
AnnotationSource |
Add an annotation source |
SqlAnnotationSourceContract |
SqlAnnotationSource |
Join a source in BigQuery |
WritableAnnotationSourceContract |
an annotation source that is written to (opt-in) | A source Altar writes to |
AllelicRecordBackendContract |
AllelicRecordBackend |
Fixtures |
VariantGeneLinkSourceContract |
VariantGeneLinkSource |
Variant-to-gene links |
VariantGeneLinkAnnotationSourceContract |
a source of the variant–gene link annotation contract | Fixtures |
VariantGeneLinkStoreContract |
VariantGeneLinkStore |
Variant-to-gene links |
StorageContract |
Storage |
Fixtures |
ConnectionContract (provisional) |
Connection |
A connection type |
PublishedDatasetContract (provisional) |
an annotation source whose publisher hosts its data | A publishing binding |
JobQueueContract (provisional) |
JobQueue |
|
AuthProviderContract (provisional) |
AuthProvider |
|
InterpretableModelContract (provisional) |
an InterpretableModel's interpretation methods and workflow |
Add interpretation to a model binding |
InterpretationStoreContract (provisional) |
InterpretationStore |
The six provisional suites test provisional contracts, and are expected to change with them.
Fixtures of other suites¶
AllelicRecordBackendContract takes a backend, a query that projects at least two fields, and the
expected_record_count it returns. ArtifactStoreContract takes a fresh store; it writes its source files to
tmp_path. VariantGeneLinkAnnotationSourceContract is AnnotationSourceContract plus the link projection rules,
so it takes the same source and known_variant_ids fixtures. assert_link_projection checks one row the same
way, for rows your own code loads elsewhere.
StorageContract takes a storage and a writable base_path. Two more fixtures are optional, and the tests that
need them are skipped when they are not overridden:
list_namesreturns a function that lists the file names directly inside a storage directory. With it, the suite checks that writes leave no temporary files behind.interrupt_writesreturns a function that takes a byte count and an exception and returns a context manager. Inside the context, each write or upload stores that many bytes of its new content, then raises the exception. With it, the suite checks that an interrupted overwrite keeps the previous file and that an interrupted first write creates no file. Supply it only if your adapter guarantees both.
LocalStorage supplies both fixtures. FsspecStorage supplies only list_names, because fsspec does not
guarantee that a failed write keeps the previous file.
Test doubles¶
altar.testing also has in-memory stand-ins for the interfaces a binding's tests touch:
| Double | Stands in for |
|---|---|
InProcessExecutionBackend (provisional) |
an ExecutionBackend: it runs each task by calling a Python function instead of a container |
InMemoryScoreStore |
a ScoreStore |
InMemoryDetailStore (provisional) |
a DetailStore |
InMemoryAnnotationSource |
an AnnotationSource |
InMemorySqlAnnotationSource |
a SqlAnnotationSource |
InMemoryAllelicRecordBackend |
an AllelicRecordBackend |
InMemoryVariantGeneLinkSource |
a VariantGeneLinkSource |
InMemoryJobQueue (provisional) |
a JobQueue |
FakeBigQueryClient (provisional) |
a BigQuery client, recording what it was asked to run |
InProcessExecutionBackend lets a container binding test run_scoring() end to end without Docker. The function
it calls stands in for the runtime: it reads the staged inputs and writes the runtime's result files.
Add a model binding shows it.
Use a contract¶
import pytest
from altar.testing import AnnotationSourceContract
class TestMySource(AnnotationSourceContract):
@pytest.fixture
def source(self):
return MySource(FakeBackend())
Read the selected contract's fixture requirements and supply only those fixtures. An annotation source that
implements a published AnnotationContract also returns it from the contract fixture, and the suite checks
the source against it with assert_implements_contract. An annotation source that is also written to, such as
a cache of annotations your own pipeline computes, additionally subclasses WritableAnnotationSourceContract.
It checks that writes are stored under the source's identity, that a source with another identity sees none of
them as annotated, and that reads never mix identities. Keep these tests in the extension package, so each Altar
version you support is checked against them.
ContractModelPlugin and contract_model_identity() are available when a store or backend test needs the same
neutral schema or a complete deterministic run identity. Model bindings test their real plugin instead.
A model binding runs PluginDiscoveryContract, ModelResultContract, and either InlineScorerContract or
ContainerScorerContract, plus one test that scores through run_scoring(). An InterpretableModel also runs
InterpretableModelContract. Annotation sources, variant–gene link sources, execution backends and stores use the
suite of their own interface.
Suites the shipped bindings run¶
| Binding | Suites |
|---|---|
altar-alphagenome |
discovery, results, inline scorer |
altar-borzoi, altar-enformer, altar-legnet, altar-sei |
discovery, results, container scorer |
altar-cherimoya, altar-chrombpnet |
discovery, results, container scorer, interpretable model |
altar-alphamissense, altar-gpnstar, altar-spliceai |
discovery, annotation source |
altar-clinvar, altar-popfreq |
discovery, annotation source, SQL annotation source, published dataset |
altar-conservation |
discovery, annotation source, published dataset |
altar-e2g-atlas |
discovery, variant–gene link source |
altar-opentargets-e2g |
discovery, variant–gene link source, variant–gene link annotation source, published dataset |
Add tests of your own¶
Conformance proves shared behavior, not scientific correctness. Also test:
- exact schemas, units, and field direction;
- supported and rejected variants;
- missing, zero, null, and duplicate records;
- provenance and genome-build mismatches;
- deterministic task commands and resource requests;
- importability without optional SDKs, credentials, GPUs, or weights;
- entry-point discovery in an installed environment.
Type-check a binding against Altar's public modules only, and test it in a clean environment that holds only the package and its declared dependencies.
Next¶
- Maturity labels lists the provisional suites and test doubles.
- Choose an interface lists every interface these suites test.