Conformance suites¶
altar.testing holds the conformance suites and test doubles an extension package runs in its own tests.
Validate an implementation shows how to use them.
Stable, except the suites for provisional contracts (AuthProviderContract, ConnectionContract,
JobQueueContract, PublishedDatasetContract, InterpretationStoreContract, InterpretableModelContract) and the
test doubles InMemoryJobQueue, InProcessExecutionBackend, InMemoryDetailStore and FakeBigQueryClient.
Provisional names lists them.
InProcessExecutionBackend runs each task through a Python function that stands in for the model's runtime, and
InMemoryDetailStore keeps detail rows in memory. Together they run run_scoring end to end in a test; the
inline model guide shows how.
testing
¶
Stable conformance suites and fixtures for external Altar extensions.
Import a contract and subclass it to prove your adapter satisfies a protocol — the same suites core runs
against its own reference adapters. Pytest is a base dependency so this stable façade is always importable;
the test extra adds the remaining tools used by Altar's own suite.
from altar.testing import StorageContract
class TestMyStorage(StorageContract):
@pytest.fixture
def storage(self):
return MyStorage(...)
@pytest.fixture
def base_path(self, tmp_path):
return str(tmp_path)
AuthProviderContract, JobQueueContract, and InMemoryJobQueue belong to the provisional
authentication and job-queue contracts, ConnectionContract to the provisional named-connection contract, and
PublishedDatasetContract to the provisional published-dataset contract, and check_registry and
assert_valid_registry, which check a registry of models and genomes, to the provisional registry format. They
are listed in __provisional__ and are expected to change with those contracts.
FakeBigQueryClient, a recording stand-in for a BigQuery client, is provisional as well, and so are the test
doubles InProcessExecutionBackend, which runs each task by calling a Python function instead of a container,
and InMemoryDetailStore. InterpretationStoreContract and InterpretableModelContract check the
provisional interpretation store and interpretable-model contracts. render_bucket_exports renders every
result-bucket export layout through a storage, and compare_wide_tsvs (with WideTsvComparison and
CellDifference) compares two wide downloads row by row, keyed by variant, and compare_prioritized_ndjson
(with NdjsonComparison and FieldDifference) two prioritized NDJSON exports record by record; all belong to
the provisional result-bucket export.
CONTRACT_ARTIFACT
module-attribute
¶
CONTRACT_ARTIFACT = ArtifactSchema(
name="profile_matrix",
version="1.0.0",
description="Synthetic backend-neutral per-batch matrix used to verify artifact stores.",
media_type="application/x-npz",
)
The artifact ArtifactStoreContract declares for ContractModelPlugin's run identity.
CONTRACT_VARIANT_ARTIFACT
module-attribute
¶
CONTRACT_VARIANT_ARTIFACT = ArtifactSchema(
name="variant_tracks",
version="1.0.0",
description="Synthetic backend-neutral per-variant track file used to verify artifact stores.",
media_type="application/x-npz",
scope=ArtifactScope.VARIANT,
)
The variant-scoped artifact ArtifactStoreContract declares: one file per variant of a task's batch.
CONTRACT_MODEL_ANNOTATIONS
module-attribute
¶
CONTRACT_MODEL_ANNOTATIONS = AnnotationContract(
source_id="org.kundajelab.altar.testing.annotations",
schema_version="1.0.0",
columns=(
AnnotationColumn(
name="region_class",
dtype="str",
label="Region class",
),
),
)
The synthetic annotation contract ContractModelPlugin's prioritization depends on (region_class).
CONTRACT_MODEL_GENOME_BUILD
module-attribute
¶
The synthetic assembly contract_model_identity scores on; a conformance source declares it to be joined.
AllelicRecordBackendContract
¶
Reusable checks for a source-agnostic allelic record backend.
Subclasses provide backend, a known query, and expected_record_count. The query should
project at least two fields and may match multiple records for one key.
InMemoryAllelicRecordBackend
¶
InMemoryAllelicRecordBackend(
rows: Sequence[InMemoryAllelicRow],
)
A projection-aware backend double shared by annotation binding tests.
InMemoryAllelicRow
dataclass
¶
InMemoryAllelicRow(
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
fields: Mapping[str, object],
)
One physical allelic key plus opaque source-native fields.
AnnotationSourceContract
¶
Shared checks that every AnnotationSource implementation must pass.
To use it, subclass this and provide two fixtures:
source— a ready-to-useAnnotationSourceinstance. A source that holds a connection is yielded and closed after the test; the suite checks thatclosecan be called twice.known_variant_ids— variant IDs the source has annotations for.
Override the optional contract fixture to return the AnnotationContract the source implements; the
suite then checks it with assert_implements_contract.
contract
¶
contract() -> AnnotationContract | None
Return the AnnotationContract the source implements, or None (the default) when it declares none.
test_columns_are_annotation_columns
¶
Column declarations have unique safe names, logical types, and labels.
test_identity_is_valid_or_none
¶
test_identity_is_valid_or_none(
source: AnnotationSource,
) -> None
identity() returns None or a valid AnnotationSourceIdentity that survives a round trip.
test_identity_build_is_one_the_class_supports
¶
test_identity_build_is_one_the_class_supports(
source: AnnotationSource,
) -> None
The assembly identity() names is among the class's supported_genome_builds, when the class declares them.
test_option_descriptions_name_its_options
¶
test_option_descriptions_name_its_options(
source: AnnotationSource,
) -> None
Each key of the class's option_descriptions names a keyword parameter of its from_options.
test_implements_declared_contract
¶
test_implements_declared_contract(
source: AnnotationSource,
contract: AnnotationContract | None,
) -> None
When the contract fixture names a contract, the source implements it.
test_implements_its_class_contract
¶
test_implements_its_class_contract(
source: AnnotationSource,
) -> None
When the source's class declares annotation_contract(), the instance implements that contract.
test_prioritize_predicate_is_predicate_or_none
¶
prioritize_predicate() returns a Predicate or None.
test_annotate_is_keyed_by_variant_id
async
¶
annotate() keys its result by variant ID, with a dict value for each variant.
test_annotate_omits_unknown_variant
async
¶
A variant the source has no row for is left out of the annotate() result.
test_output_feeds_merge_annotations
async
¶
annotate() output works as an input to merge_annotations.
InMemoryAnnotationSource
¶
InMemoryAnnotationSource(
name: str,
columns: list[AnnotationColumn],
rows: Mapping[str, Mapping[str, Any]],
prioritize_predicate: Predicate | None = None,
*,
identity: AnnotationSourceIdentity | None = None,
)
Bases: AnnotationSource
An AnnotationSource backed by an in-memory dict, for use in tests.
rows maps a variant ID to its {column: value} annotations. The source is read-only, like the real
reference-data sources it stands in for, so it keeps the base class's read-only add_annotations.
identity is what identity() returns; pass one to stand in for a source that implements a contract.
ArtifactStoreContract
¶
Check provenance, content addressing, coverage, lineages, idempotency, filtering, and schema collisions.
Subclass this and provide a fresh store fixture. A store that holds a connection is yielded and closed
after the test; the suite checks that close can be called twice. Source files are written to tmp_path
and offered through LocalStorage, as the scoring engine offers a collected output; the store copies them
wherever it keeps bytes.
lineage_id
staticmethod
¶
lineage_id(identity: PluginRunIdentity) -> str
Return the score lineage a run of identity writes under, as its primary rows carry it.
source
staticmethod
¶
Write one artifact file and return its URI and digest, as the engine hands them to a store.
test_lineages_coexist_and_reads_accept_named_lineages
async
¶
test_lineages_coexist_and_reads_accept_named_lineages(
store: ArtifactStore,
schema: ArtifactSchema,
identity: PluginRunIdentity,
tmp_path: Path,
) -> None
Records of two lineages coexist under one model; a read counts only the lineages it accepts.
test_accepted_lineage_with_another_schema_version_does_not_supply_files
async
¶
test_accepted_lineage_with_another_schema_version_does_not_supply_files(
store: ArtifactStore,
schema: ArtifactSchema,
identity: PluginRunIdentity,
tmp_path: Path,
) -> None
An accepted lineage's artifact stands in only when it is the same artifact at the same version.
test_variant_scoped_files_each_cover_and_list_one_variant
async
¶
test_variant_scoped_files_each_cover_and_list_one_variant(
store: ArtifactStore,
identity: PluginRunIdentity,
tmp_path: Path,
) -> None
A task's per-variant files are one record each, keyed and listed by their variant.
AuthProviderContract
¶
Checks that an AuthProvider implementation behaves as expected.
Subclass this and provide a provider fixture. If verification depends on the token contents, also
override the valid_token fixture to return a token the provider accepts. The default token is an
opaque string that no-auth providers ignore.
FakeBigQueryClient
¶
Records calls and returns programmed score and annotation responses.
ConnectionContract
¶
Shared checks that every Connection type must pass.
Subclass it and provide connection_type. Override the optional fixtures to describe the type:
connection_options— run spec options that build a working connection (default{}).connection— the connection under test; by defaultconnection_type.from_options(**connection_options). Override it to install fakes for a provider SDK first.served_uri— a URI the connection reaches object storage for, orNoneif it reaches none.unsupported_uri— a URI whose scheme it must decline, orNoneif it serves every scheme.reaches_bigquery— whetherbigquery_client()returns a client (defaultFalse).credential_options— spec options that carry a credential, whichfrom_optionsmust refuse.connection_holding_secret— a connection a host built in code with a secret, and the secret, orNone.
credential_options
¶
Spec options that carry a credential; by default fsspec-style keys a provider reads as a secret.
test_a_connection_reaches_a_published_copy_without_changing_its_identity
¶
test_a_connection_reaches_a_published_copy_without_changing_its_identity(
connection: Connection,
served_uri: str | None,
reaches_bigquery: bool,
) -> None
PublishedDataset.select_copy reaches each copy through this connection and leaves the identity alone.
Every binding that publishes its data resolves its run spec options with select_copy, so this is the
path a connection takes into a source; the bindings' PublishedDatasetContract then checks, by reading,
that a source built through a connection declares the same identity as one built without.
ContractModelPlugin
¶
DetailStoreContract
¶
Check provenance, logical keys, parent presence, lineages, idempotency, filtering, and schema collisions.
Subclass this and provide a fresh store fixture. A store that holds a connection is yielded and closed after
the test; the suite checks that close can be called twice.
lineage_id
staticmethod
¶
lineage_id(identity: PluginRunIdentity) -> str
Return the score lineage a run of identity writes under, as its primary rows carry it.
test_lineages_coexist_and_reads_accept_named_lineages
async
¶
test_lineages_coexist_and_reads_accept_named_lineages(
store: DetailStore,
schema: DetailSchema,
identity: PluginRunIdentity,
) -> None
Rows of two lineages coexist under one model; a read counts only the lineages it accepts.
test_reads_take_each_variants_rows_from_the_first_accepted_lineage_with_rows
async
¶
test_reads_take_each_variants_rows_from_the_first_accepted_lineage_with_rows(
store: DetailStore,
schema: DetailSchema,
identity: PluginRunIdentity,
) -> None
With several accepted lineages, each variant's rows come from one lineage, as primary reads choose one.
It is the first lineage, in the read's preference order, that holds rows for the variant. Rows of two producers never mix, even when they have different logical keys.
test_rejects_non_finite_values
async
¶
test_rejects_non_finite_values(
store: DetailStore,
schema: DetailSchema,
identity: PluginRunIdentity,
) -> None
A NaN or infinite value is refused on write, and the call stores none of its rows.
test_accepted_lineage_with_another_schema_version_does_not_supply_rows
async
¶
test_accepted_lineage_with_another_schema_version_does_not_supply_rows(
store: DetailStore,
schema: DetailSchema,
identity: PluginRunIdentity,
) -> None
An accepted lineage's table stands in only when it is the same table at the same version.
InMemoryDetailStore
¶
Bases: DetailStore
A DetailStore that keeps every table in a dict, for tests of bindings and drivers.
It behaves as SqliteDetailStore does: tables are keyed by model, detail name and lineage, rows by their
logical key, and a second write of a key is skipped. Rows are stored as their canonical JSON, so a read
returns copies with the same values a SQLite store returns, and a NaN or infinite value is refused before
any row of the call is stored.
ExecutionBackendContract
¶
Checks that an ExecutionBackend implementation behaves as expected.
Subclass this, provide a backend fixture that returns an instance ready to submit and poll, and
implement finish_task, expire_task, and launched_timeout against the fake runtime the backend is
wired to. A backend whose input_digest_verification is "task" also implements
fail_input_verification. A backend whose failure_detail reads task logs implements write_task_log.
A backend that declares adopts_active_tasks or clears_outputs_on_fresh_start implements
write_task_output and task_output, and start_task when its tasks clear their own outputs as they start.
Override the spec fixture if the default trivial task cannot run on your
backend, and set unlimited_timeout_s if your platform cannot run a task without a time limit.
finish_task
¶
finish_task(
backend: ExecutionBackend,
handle: TaskHandle,
*,
exit_code: int,
) -> None
Script the fake runtime so the task behind handle has exited with exit_code.
expire_task
¶
expire_task(
backend: ExecutionBackend, handle: TaskHandle
) -> None
Script the fake runtime so the task behind handle has run past its timeout_s, as the platform
would show it: for example a failed Job with a deadline condition, a timeout exit code, or a
container that is still running long after it started.
launched_timeout
¶
launched_timeout(
backend: ExecutionBackend, handle: TaskHandle
) -> int | None
Return the effective deadline, in seconds, the platform would enforce for the task behind handle,
or None if it would enforce none. Report what the platform applies, not only what the fake received:
if the provider SDK substitutes a default when no deadline is passed, return that default.
write_task_log
¶
write_task_log(
backend: ExecutionBackend,
handle: TaskHandle,
lines: list[str],
) -> bool
Script the fake runtime so the task behind handle has written lines to its log, and return True.
The default returns False, for a backend whose failure_detail reads no log.
fail_input_verification
¶
fail_input_verification(
backend: ExecutionBackend, handle: TaskHandle
) -> None
Script the fake runtime so the task behind handle failed its in-task input digest check. Only a
backend whose input_digest_verification is "task" needs this.
write_task_output
¶
write_task_output(
backend: ExecutionBackend,
transfer: Transfer,
content: bytes,
) -> None
Write content where a task of backend writes the output transfer, as a task would.
Only a backend that declares adopts_active_tasks or clears_outputs_on_fresh_start needs this.
Provisional: the hook is expected to change.
task_output
¶
task_output(
backend: ExecutionBackend, transfer: Transfer
) -> bytes | None
Return the bytes at the place a task of backend writes the output transfer, or None if none.
Only a backend that declares adopts_active_tasks or clears_outputs_on_fresh_start needs this.
Provisional: the hook is expected to change.
start_task
¶
start_task(
backend: ExecutionBackend, handle: TaskHandle
) -> None
Script the fake runtime so the task behind handle has started.
It runs any step the platform runs before the task's command, such as an init container that clears its outputs.
The default does nothing, for a backend whose submit clears outputs before it launches the task.
Provisional: the hook is expected to change.
test_resubmitting_an_active_spec_does_not_run_it_twice_at_once
async
¶
test_resubmitting_an_active_spec_does_not_run_it_twice_at_once(
backend: ExecutionBackend, spec: ContainerTaskSpec
) -> None
A driver that restarts mid-run submits a spec again while its first task is still active. The second
submit must not raise. If it returns the first task (the same external_id), that handle must report
the first task's outcome without being driven itself, so the work is not run twice at once. A backend
that cannot recognize the earlier task launches a separate one, whose state must be its own.
test_resubmitting_after_a_terminal_outcome_runs_the_task_again
async
¶
test_resubmitting_after_a_terminal_outcome_runs_the_task_again(
backend: ExecutionBackend,
spec: ContainerTaskSpec,
exit_code: int,
) -> None
Once a task has finished, succeeded or failed, submitting its spec again runs it again. The new handle must not report the old outcome: a spec names its inputs by path, and a later run may have put new content there.
test_input_digest_on_a_directory_is_rejected
async
¶
test_input_digest_on_a_directory_is_rejected(
backend: ExecutionBackend,
spec: ContainerTaskSpec,
tmp_path: Path,
) -> None
A digest covers the bytes of one regular file, so a digest-bearing directory is rejected exactly like a mismatch rather than hashed by some backend-specific tree rule.
test_an_adopted_task_keeps_the_outputs_it_wrote
async
¶
test_an_adopted_task_keeps_the_outputs_it_wrote(
backend: ExecutionBackend,
spec: ContainerTaskSpec,
tmp_path: Path,
) -> None
A driver that restarts submits a running task's spec again. A backend that adopts that task must keep the outputs it already wrote, or the task finishes without them.
CellDifference
dataclass
¶
One cell that differs between two wide TSVs: its row's variant_id, its column, and both values.
FieldDifference
dataclass
¶
One field that differs between two prioritized NDJSON records (provisional).
path names it from the record's top: am_class, or model_scores[m1].logfc for a model's entry.
NdjsonComparison
dataclass
¶
NdjsonComparison(
only_left: tuple[str, ...] = (),
only_right: tuple[str, ...] = (),
differences: tuple[FieldDifference, ...] = (),
)
The result of compare_prioritized_ndjson (provisional).
only_left and only_right are the variants one side has and the other lacks, and differences every field
whose values differ. matches holds when both sides hold the same variants with equal values.
WideTsvComparison
dataclass
¶
WideTsvComparison(
same_header: bool,
only_left: tuple[str, ...] = (),
only_right: tuple[str, ...] = (),
differences: tuple[CellDifference, ...] = (),
allowed: tuple[CellDifference, ...] = (),
)
The result of compare_wide_tsvs.
same_header says the headers are identical. only_left and only_right are the variant_ids one file has
and the other lacks. differences are the differing cells outside allowed_columns, and allowed the
differing cells inside them. matches holds when the headers match, both files hold the same variants, and
no cell outside the allowed columns differs.
InProcessExecutionBackend
¶
InProcessExecutionBackend(
runtime: Callable[[ContainerTaskSpec], int | None]
| None,
*,
data_root: str | PathLike[str],
storage: Storage | None = None,
)
Bases: ExecutionBackend
Runs each task of a scoring plan by calling a Python function in place of the model's container.
The function stands in for the model runtime. It takes the task's ContainerTaskSpec, reads the staged inputs
the command names, writes the outputs the task declares, and returns the exit code (None means 0). Build the
ScoringRequest with this backend's path_resolver(), so the command names real files under data_root.
A runtime that parses its command the way the model's CLI does is a close stand-in for the image.
submit calls the function and records its outcome; the next poll reports it. What the function prints, and
the traceback of an exception it raises, become the task's log, which failure_detail returns. An exception
or SystemExit fails the task with the matching exit code (1 for an exception).
Without a runtime (runtime=None), a submitted task runs until a test finishes it with finish or expire.
Use that to test a driver's handling of slow, failed or timed-out tasks.
Every submit starts a fresh task. It first deletes the file at each declared output under data_root, as the
local backend does, so a task never hands back a file an earlier task left there.
stage_inputs copies each input into data_root at its logical path and checks its digest there, as the
local backend does. collect_outputs copies each output from data_root to its URI. A plan names the
variant file and result files by their logical path, so pass on_data_root as run_scoring's
input_transfer and output_transfer, and write the variant file at host_path(...).
The backend keeps what it was asked to do for a test to check: submitted (every spec, in order),
staged (every staged input) and cancelled (the external IDs of cancelled tasks).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
runtime
|
Callable[[ContainerTaskSpec], int | None] | None
|
The function that runs a task, or |
required |
data_root
|
str | PathLike[str]
|
The directory that stands in for the volume every task reads and writes. |
required |
storage
|
Storage | None
|
The |
None
|
path_resolver
¶
path_resolver() -> PathResolver
Return the resolver that maps a logical path to its file under data_root.
host_path
¶
Return the file under data_root that a task reads or writes at logical_path.
on_data_root
¶
Point a transfer that names a layout file at its file under data_root; return any other unchanged.
A plan gives the variant file and the result files their logical path as URI, while a resource keeps the
URI it was configured with. Pass this as run_scoring's input_transfer and output_transfer: the
engine then reads the variant file where you wrote it, and the results where the runtime wrote them.
writes_output_in_place
¶
writes_output_in_place(transfer: Transfer) -> bool
Whether transfer.uri is the file under data_root the task writes, rather than where collect_outputs
copies it.
stage_inputs
async
¶
stage_inputs(transfers: list[Transfer]) -> None
Copy each input to its logical path under data_root and check its digest there.
A copy that does not match its digest is deleted before TransferIntegrityError is raised. An input
already at its place under data_root is checked where it is and never deleted.
collect_outputs
async
¶
collect_outputs(transfers: list[Transfer]) -> None
Copy each output from its logical path under data_root to its URI.
An output the task did not write is skipped, so the scoring engine reports it as not produced.
submit
async
¶
submit(spec: ContainerTaskSpec) -> TaskHandle
Record the task and, when the backend has a runtime, run it now.
The file at each declared output is deleted first, and its directory created, so the runtime writes its files directly and a file an earlier task left there is never collected. A directory at an output's path is left alone.
poll
async
¶
poll(handles: list[TaskHandle]) -> list[TaskStatus]
Return each task's recorded state. A task this backend never ran is missing.
cancel
async
¶
cancel(handle: TaskHandle) -> None
Record the cancellation. A task that has not finished then reports reason cancelled.
failure_detail
async
¶
failure_detail(handle: TaskHandle) -> str | None
Return the last 20 lines of the task's log, or None when it wrote none or is unknown.
spec
¶
spec(handle: TaskHandle) -> ContainerTaskSpec
Return the spec the task behind handle was submitted with.
finish
¶
finish(handle: TaskHandle, *, exit_code: int = 0) -> None
Mark the task behind handle as exited with exit_code.
expire
¶
expire(handle: TaskHandle) -> None
Mark the task behind handle as having run past its timeout_s.
The backend records each spec's timeout_s but does not enforce it, since a runtime call cannot be
interrupted.
write_log
¶
write_log(handle: TaskHandle, lines: Iterable[str]) -> None
Append lines to the log of the task behind handle.
InterpretableModelContract
¶
Check a binding's interpretation methods, descriptors, workflows and rows.
Subclass this and provide the fixtures: plugin; request, an interpretation request for a fully
configured model; selection, the method and target whose chain covers every method; golden_ids,
the pinned interpretation ID of each method of that chain; and variants, canonical keys whose windows lie
inside the reference.
InterpretationStoreContract
¶
Check declaring, putting, committing and reading interpretation products.
Subclass this and provide a fresh store fixture. Source files are written to tmp_path and offered
through LocalStorage, as ingest offers a collected output.
declare
async
staticmethod
¶
declare(
store: InterpretationStore,
operation: InterpretationOperation,
) -> None
Record operation and declare its products: contributions and profiles upstream, hits downstream.
put
async
staticmethod
¶
put(
store: InterpretationStore,
tmp_path: Path,
descriptor: InterpretationDescriptor,
product: str,
run_id: str,
*,
variants: Sequence[str] = CONTRACT_VARIANTS,
shard: tuple[tuple[str, str], ...] = (),
content: bytes | None = None,
model_id: str = "m1",
) -> InterpretationRecord
Write one product file and put it under descriptor's ID in execution run_id.
put_upstream
async
¶
put_upstream(
store: InterpretationStore,
tmp_path: Path,
operation: InterpretationOperation,
run_id: str,
) -> list[InterpretationRecord]
Put the upstream method's three files: contributions for each allele and one profile file.
commit_chain
async
¶
commit_chain(
store: InterpretationStore,
tmp_path: Path,
operation: InterpretationOperation,
run_id: str,
) -> tuple[
tuple[InterpretationRecord, ...],
tuple[InterpretationRecord, ...],
]
Put and commit both methods of operation in execution run_id.
InMemoryJobQueue
¶
Bases: JobQueue
A JobQueue that records enqueued jobs in a list, used to run the conformance suite.
It is constructed with the set of kind values it accepts. enqueue records the (kind, parameters)
call and returns an opaque run-<n> handle; a kind it was not told about raises KeyError,
satisfying the ABC's "raise on a kind it does not recognize" contract. It is not registered as an
entry-point adapter.
JobQueueContract
¶
Shared test suite that every JobQueue adapter must pass.
To run it against your adapter, subclass this and add a job_queue fixture (a fresh adapter) and a
valid_kind fixture (a kind the adapter accepts). Set unknown_kind_error if the adapter raises a
type more specific than Exception for an unrecognized kind.
ModelPluginContract
¶
Bases: ModelResultContract
Backward-compatible name for the result/schema-only contract.
Compute capability checks intentionally live in InlineScorerContract and
ContainerScorerContract; this class is not an all-in-one model contract.
ModelResultContract
¶
Checks the result/schema capability of a ModelPlugin.
Subclass this and provide a plugin fixture that returns the plugin under test.
test_variant_eligibility_is_consistent
¶
test_variant_eligibility_is_consistent(
plugin: ModelPlugin,
) -> None
Check that the declared eligibility is well-formed and serializes, not that it matches the runtime.
This cannot catch a wrong declaration, such as an SNV-only runtime whose manifest admits indels. A binding guards that with its own test of the classes its runtime accepts.
PluginDiscoveryContract
¶
Check one external plugin's metadata-only listing and targeted validated load.
PublishedDatasetContract
¶
Shared checks for a binding that publishes its data and reads it in three modes.
Provide these fixtures:
source_class— the binding'sAnnotationSourcesubclass, registered under itsname, whosepublished_dataset()returns the published release and whosefrom_optionsreads the three modes.built_dataset— the local directory of a small dataset the binding's builder wrote.built_release_fields— the footer fields the binding declares for that dataset's release, computed as its published dataset'srelease_fieldsare.known_variant_ids— canonical variant IDs the built dataset has rows for.
Optional: published (default source_class.published_dataset()) and reads_bigquery_tables (default
True). A binding with no table mode returns False; its from_options then refuses table and
release with a ValueError that names the option.
owned_source
¶
owned_source(
source_class: type[AnnotationSource],
built_dataset: str | PathLike[str],
) -> AnnotationSource
Return the source over the built dataset, read as an owned Parquet copy.
stand_in
¶
stand_in(
published: PublishedDataset,
built_dataset: str | PathLike[str],
built_release_fields: dict[str, str],
owned_source: AnnotationSource,
) -> Iterator[PublishedDataset]
Publish the built dataset at in-memory locations named like the real ones; remove it afterwards.
published_stand_in
¶
published_stand_in(
source_class: type[AnnotationSource],
stand_in: PublishedDataset,
monkeypatch: MonkeyPatch,
) -> PublishedDataset
Point the source's published_dataset() at the stand-in, as if its publisher had published the build.
InMemoryScoreStore
¶
Bases: ScoreStore
A ScoreStore that keeps everything in a dict, used to run the conformance suite and the
composition tests. It implements the persistence methods (including read_score) and inherits
materialize from the base class.
Each row is a Mapping[str, Any] with variant_id plus the plugin's score columns. Scores are keyed
by (model_id, variant_id). Re-inserting the same key does nothing, and add_scores counts it under
InsertResult.skipped.
read_score
async
¶
Return one model's stored score for a variant, or None if there isn't one.
This store implements only read_score and inherits the base class's read_scores, which reads
each score by calling this method in a loop. The conformance suite therefore covers that default
read_scores, while SqliteScoreStore covers a bulk override.
ScoreStoreContract
¶
Shared test suite that every ScoreStore implementation must pass.
To run it against your store, subclass this and add a store fixture that returns a fresh, empty
store instance. A store that holds a connection is yielded and closed after the test. Each test method below
checks one behavior the store must have, including that close can be called twice.
creates_descriptor_table_on_write
class-attribute
¶
Whether the lineage store creates its descriptor sidecar when it first records a descriptor (SQLite does), rather than requiring it to be provisioned (BigQuery).
make_annotation_source
¶
make_annotation_source() -> Callable[..., AnnotationSource]
Return a factory for the annotation sources the materialize tests join.
The factory takes name, columns, rows ({variant_id: {column: value}}), an optional
prioritize_predicate, and identity, the AnnotationSourceIdentity the source declares; these are the
arguments of InMemoryAnnotationSource. The suite always passes an identity on the contract model's
assembly, CONTRACT_MODEL_GENOME_BUILD, unless a test is about another assembly, since a source that
declares none serves only hg38. Override it when the store can only join sources that live on its own
backend.
contract_source
¶
contract_source(
make_annotation_source: Callable[..., AnnotationSource],
) -> Callable[..., AnnotationSource]
Return make_annotation_source with each source declaring that it serves the contract model's assembly.
Each source declares a contract of its own, named after it, so a model's dependency still resolves its columns by name.
lineage_store
¶
Return a fresh, empty store in compact-lineage mode; override it when the store has one.
descriptor_table_exists
¶
descriptor_table_exists() -> Callable[[ScoreStore], bool]
Return a probe for whether a lineage_store's descriptor sidecar exists; override it with the mode.
provision_descriptor_table
¶
provision_descriptor_table() -> Callable[
[ScoreStore], Awaitable[None]
]
Return a coroutine function that provisions a lineage_store's descriptor sidecar as a host would,
with one extra nullable column; override it with the mode.
test_ensure_schema_accepts_plugin_columns
async
¶
ensure_schema accepts the plugin's columns without raising.
test_add_scores_reports_rows_added
async
¶
add_scores returns an InsertResult reporting how many rows it appended.
test_primary_write_unit_commits_every_batch
async
¶
test_primary_write_unit_commits_every_batch(
store: ScoreStore, columns
) -> None
A cleanly exited primary_write unit persists every batch its writer accepted, with per-batch counts.
test_primary_write_unit_aborts_on_error
async
¶
test_primary_write_unit_aborts_on_error(
store: ScoreStore, columns
) -> None
An exception inside the unit propagates; a store that claims atomicity then holds none of its rows.
A pass-through store (atomic_primary_write = False) keeps the batches it already added. Either way the
store accepts a new unit afterwards.
test_lineage_primary_write_unit_commits_or_aborts_as_a_whole
async
¶
test_lineage_primary_write_unit_commits_or_aborts_as_a_whole(
lineage_store: ScoreStore, columns
) -> None
In compact-lineage mode an atomic unit's abort also leaves no lineage provenance behind.
test_get_unscored_excludes_scored_variants
async
¶
get_unscored_variants returns only the variants not yet scored for the model.
test_non_finite_scores_are_refused_on_write
async
¶
A NaN or infinite score raises ValueError from add_scores and from a write unit, and nothing is stored.
A missing score is None; NaN would read back as NULL from one store and as NaN from another.
test_nullable_scores_round_trip_as_null
async
¶
A None in every nullable score column reads back as None, never as NaN, False, or an empty string.
test_duplicate_writes_read_back_as_one_whole_row
async
¶
Two writes of one (model_id, variant_id) read back as one of the two rows, whole.
A store may skip the second write or keep both rows. A read must never mix two rows' columns, such as one
row's effect with the other's quality_rank. Each row leaves the other's values null, so a store that
picks each column's first non-null value would mix them.
test_materialize_source_predicate_matches_quotes_backslashes_and_newlines
async
¶
test_materialize_source_predicate_matches_quotes_backslashes_and_newlines(
store, contract_source
) -> None
A prioritizing source's string literals match values holding a quote, a backslash, or a newline.
A store that runs the predicate in SQL must escape each literal for its own dialect; GoogleSQL, for one, reads a doubled quote as a syntax error and a backslash as an escape.
ContainerScorerContract
¶
Check labels, resources, transfers, outputs, identity, and serialization.
test_scoring_request_round_trip_preserves_the_layout
¶
test_scoring_request_round_trip_preserves_the_layout(
plugin: ModelPlugin, scoring_request: ScoringRequest
) -> None
A durable request must rebuild the same layout class and field values the plan was built from.
The envelope records the layout's fields and the plugin names the class, so a binding layout subclass does not revert to the model-neutral default after a restart.
InlineScorerContract
¶
Check a model's declared in-process scoring capability and returned rows.
test_with_run_variants_carries_exactly_those_variants
async
¶
test_with_run_variants_carries_exactly_those_variants(
plugin: ModelPlugin,
scoring_request: ScoringRequest,
inline_runtime: RuntimeContext | None,
) -> None
Check the hook prepare_scoring_batches binds each dense batch through.
It must change only the run inputs, leave the scientific identity alone, and score exactly the variants it was given, in order.
InMemorySqlAnnotationSource
¶
InMemorySqlAnnotationSource(
name: str,
columns: list[AnnotationColumn],
rows: Mapping[str, Mapping[str, Any]],
prioritize_predicate: Predicate | None = None,
)
Bases: InMemoryAnnotationSource, SqlAnnotationSource
An in-memory AnnotationSource that also advertises the SQL-fusion capability, for tests.
It reuses InMemoryAnnotationSource's dict backing for annotate() and adds a sql_join that returns a
synthetic JoinSpec projecting exactly its declared columns. The relation is a placeholder table
reference — nothing executes it — so the double exercises the contract's shape invariants without a real
SQL backend.
SqlAnnotationSourceContract
¶
Bases: AnnotationSourceContract
Shared checks a SqlAnnotationSource must pass, on top of the base AnnotationSource contract.
Subclass this and provide the base suite's source (a SqlAnnotationSource instance) and
known_variant_ids fixtures. The sql_join_context fixture below can be overridden if a source needs a
non-default context.
test_source_is_sql_annotation_source
¶
The source advertises the capability by subclassing SqlAnnotationSource (how a store detects it).
test_sql_join_returns_joinspec_or_none
¶
sql_join returns a JoinSpec (fuse) or None (fall back to annotate()).
test_sql_join_uses_context_join_key
¶
A returned JoinSpec joins on the context's join_key. The store builds
ON <alias>.<spec.join_key> = <base>.<spec.join_key>, so a spec that picked a different key would
reference a column the base relation does not expose.
test_sql_join_projects_only_declared_columns
¶
A returned JoinSpec has a non-empty alias/relation and projects only columns the source declares
in annotation_columns(). The store adds <alias>.<col> for each, and the materialized schema is built
from annotation_columns(), so a projected column the source doesn't declare would have no schema slot.
StorageContract
¶
Behavior every Storage implementation must satisfy.
Subclasses must provide two fixtures:
storage— a ready-to-useStorageinstance.base_path— a writable path or prefix the storage understands (a temp dir for local disk, a bucket prefix for object storage). The suite builds paths by joining onto it with/.
Subclasses may also override two optional fixtures. Each returns None by default, which skips the tests
that need it:
list_names— a function that takes a storage directory path and returns the names of the files directly inside it. The suite uses it to check that writes leave no temporary files.interrupt_writes— a function that takes a byte count and an exception and returns a context manager. Inside the context, every write and upload through the adapter stores that many bytes of its new content the way the adapter normally stores data in progress, then raises the exception. Supply it only when the adapter guarantees that a failed write leaves the previous file unchanged and creates no new file.
InMemoryVariantGeneLinkSource
¶
InMemoryVariantGeneLinkSource(
links: Sequence[VariantGeneLink],
*,
error: VariantGeneLinkSourceError | None = None,
)
VariantGeneLinkSourceContract
¶
Shared checks every external VariantGeneLinkSource can run.
Subclasses provide source, a known_query with at least two expected records,
expected_record_ids in stable order, and an empty_query known to have no records.
VariantGeneLinkAnnotationSourceContract
¶
Bases: AnnotationSourceContract
AnnotationSourceContract plus the projection rules of the variant-gene link annotation contract.
Subclass it with the source and known_variant_ids fixtures. The contract fixture is the link contract,
so the source must declare its identity and columns, and every known variant's row must satisfy
assert_link_projection.
VariantGeneLinkStoreContract
¶
Checks visibility, idempotency, filtering, ordering, and lossless round trips.
WritableAnnotationSourceContract
¶
Identity checks every writable annotation lake must pass; see the module docstring for the fixtures.
test_writes_are_stamped_with_the_source_identity
async
¶
test_writes_are_stamped_with_the_source_identity(
open_source: OpenSource,
identity: AnnotationSourceIdentity,
annotation_rows: Rows,
stored_identity_hashes: StoredHashes,
) -> None
Rows are stored with the identity, and a later source of that identity serves and counts them.
test_another_identity_sees_nothing_annotated
async
¶
test_another_identity_sees_nothing_annotated(
open_source: OpenSource,
identity: AnnotationSourceIdentity,
other_identity: AnnotationSourceIdentity,
annotation_rows: Rows,
) -> None
A new release re-annotates: none of the old release's rows count as annotated or are served.
test_reads_do_not_mix_identities
async
¶
test_reads_do_not_mix_identities(
open_source: OpenSource,
identity: AnnotationSourceIdentity,
other_identity: AnnotationSourceIdentity,
annotation_rows: Rows,
other_annotation_rows: Rows,
stored_identity_hashes: StoredHashes,
) -> None
A lake holding two releases of one variant serves each source only its own release's values.
test_rows_carrying_another_identity_are_rejected
async
¶
test_rows_carrying_another_identity_are_rejected(
open_source: OpenSource,
identity: AnnotationSourceIdentity,
other_identity: AnnotationSourceIdentity,
annotation_rows: Rows,
) -> None
A row stamped with another identity's hash is never written under this one.
test_legacy_rows_are_used_only_after_an_explicit_backfill
async
¶
test_legacy_rows_are_used_only_after_an_explicit_backfill(
open_source: OpenSource,
open_legacy_source: OpenLegacySource,
identity: AnnotationSourceIdentity,
annotation_rows: Rows,
) -> None
Rows written without an identity are neither served nor counted until the backfill adopts them.
example_allelic_query
¶
example_allelic_query() -> AllelicRecordQuery
Return the small deterministic query used by the reference conformance test.
assert_implements_contract
¶
assert_implements_contract(
source: AnnotationSource, contract: AnnotationContract
) -> None
Assert that source declares contract and provides every contract column with the same type.
The source's identity() must name the contract's source_id, with the same major version and a version
no older than the contract's. Each contract column must appear in annotation_columns() with an equal
dtype, repeated flag and fields; display labels may differ. The source may declare more columns.
make_stub_sources
¶
make_stub_sources(
source_dir: str, plan: ContainerScoringPlan
) -> None
Create a placeholder source file for each distinct input declared by a plan.
source_transfers
¶
Map logical transfer paths to files under a host fixture directory.
stubify
¶
stubify(
spec: ContainerTaskSpec, resolver: PathResolver
) -> ContainerTaskSpec
Replace a task command with one that creates the same declared outputs.
contract_model_descriptor
¶
contract_model_descriptor(
runtime_identity: str = "altar-contract-runtime:v1",
*,
genome_build: str = CONTRACT_MODEL_GENOME_BUILD,
) -> LineageDescriptor
Return the lineage descriptor of contract_model_identity for the same runtime identity and assembly.
contract_model_identity
¶
contract_model_identity(
runtime_identity: str = "altar-contract-runtime:v1",
*,
genome_build: str = CONTRACT_MODEL_GENOME_BUILD,
) -> PluginRunIdentity
Return a complete synthetic scoring identity for persistence conformance tests.
genome_build is the assembly of the scored variants. A test that joins sources declaring no identity, which
serve only hg38, passes "hg38".
compare_prioritized_ndjson
¶
compare_prioritized_ndjson(
left: Ndjson, right: Ndjson
) -> NdjsonComparison
Compare two prioritized NDJSON exports record by record, by value (provisional).
Each side is gzipped NDJSON, as bytes or a local path, or a sequence of parts. Records are keyed by
variant_id, and each record's model_scores entries by model_id, so order never matters. Values compare as
values_equal compares them: an omitted key equals null (BigQuery's JSON export omits a null field, Altar writes
it), numbers compare by float64 value, an INT64 written as a JSON string equals the number, and "NaN",
"Infinity" and "-Infinity" equal the floats. Every differing field is reported. A variant_id that appears
twice on one side raises ValueError.
The left records are spilled to a scratch SQLite database keyed by variant_id and each right record is looked
up there, so memory holds a record at a time (and the differences found).
compare_wide_tsvs
¶
compare_wide_tsvs(
left: WideTsv,
right: WideTsv,
*,
allowed_columns: Iterable[str] = (),
nan_equals_empty: bool = False,
semantic: bool = False,
) -> WideTsvComparison
Compare two gzipped wide TSVs as sets of rows keyed by variant_id, cell by cell (provisional).
Each file is the bytes of a gzipped TSV or a local path to one. Row order is ignored. Every differing cell is
listed, under allowed when its column is in allowed_columns (a display rider's columns, whose ties and NULL
decisions may be broken differently by another export path) and under differences otherwise. Cells of a row
only one file has are not compared. A variant_id that appears twice in one file raises ValueError.
nan_equals_empty counts a NaN cell (NaN or nan) and an empty one as equal, for comparing with a file
written by an exporter that wrote NaN as an empty cell. Altar's TSV writes NaN, and BigQuery's CSV export
nan; semantic=True counts those two as equal.
semantic compares values rather than text, for comparing Altar's download with one BigQuery's own export
wrote, whose text differs while its values are equal (cells_equal): a number by its float64 value after
parsing, so 0.3333333333333333 (Altar, the shortest text that round-trips) equals 0.33333333333333331
(BigQuery, up to 17 digits) and 1e+15 equals 1000000000000000.0; nan, inf and -inf equal NaN,
Infinity and -Infinity; and a JSON-text cell (a repeated or struct annotation, a reason list) by its parsed
value under the same rules. Any other cell is compared as text.
Both files are read as streams: the left file's rows are spilled to a scratch SQLite database keyed by
variant_id, and each right row is looked up there, so memory holds a row at a time (and the differences
found), whatever the files' size, such as a download of 10,000 columns by 50,000 variants.
render_bucket_exports
async
¶
render_bucket_exports(
buckets: ResultBuckets, root: str, storage: Storage
) -> dict[str, bytes]
Render every bucket export layout under root through storage, and return each file's bytes by path.
The paths, relative to root: wide/ (the wide TSV and its column manifest), prioritized/ (the prioritized
wide TSV), parquet/ (the wide Parquet), ndjson/ (the prioritized NDJSON), and long-tsv/ and
long-parquet/ (the long layout in each format).
assert_valid_registry
¶
assert_valid_registry(
uri: str, *, connection: Connection | None = None
) -> None
Raise AssertionError listing every problem check_registry finds with the registry at uri.
check_registry
¶
check_registry(
uri: str,
*,
connection: Connection | None = None,
allow_private_hosts: bool = False,
) -> list[str]
Return every problem with the registry whose index is at uri, and with each entry it names; empty if none.
It reads the index and every entry document, and no other file. Each document must parse, each entry's bytes
must have the digest its path gives, and each index line must agree with its entry. Every URI must be one a
published registry may name (altar.plugins.registries.documents.uri_problem with publishing). When the
entry's binding is installed, its configuration must load. An index hosted elsewhere is read as a fetch reads
it, and its entries are read only where a fetch would read them (RegistrySource.check_uri).
assert_link_projection
¶
Assert that one variant's annotation row follows the contract's projection rules.
link_genes is a non-empty list of records with exactly the LINK_GENE_FIELDS, each naming a gene and a
link record, ordered strongest first with unrankable links last. link_count and link_gene_count count
those records and their distinct gene IDs. link_top_gene and link_top_score are either both absent, when
the first link cannot be ranked, or describe the first link; link_top_gene_symbol is present only with them.