Skip to content

Conformance suites

altar.testing holds the conformance suites and test doubles an extension package runs in its own tests. Validate an implementation shows how to use them.

Stable, except the suites for provisional contracts (AuthProviderContract, ConnectionContract, JobQueueContract, PublishedDatasetContract, InterpretationStoreContract, InterpretableModelContract) and the test doubles InMemoryJobQueue, InProcessExecutionBackend, InMemoryDetailStore and FakeBigQueryClient. Provisional names lists them.

InProcessExecutionBackend runs each task through a Python function that stands in for the model's runtime, and InMemoryDetailStore keeps detail rows in memory. Together they run run_scoring end to end in a test; the inline model guide shows how.

testing

Stable conformance suites and fixtures for external Altar extensions.

Import a contract and subclass it to prove your adapter satisfies a protocol — the same suites core runs against its own reference adapters. Pytest is a base dependency so this stable façade is always importable; the test extra adds the remaining tools used by Altar's own suite.

from altar.testing import StorageContract

class TestMyStorage(StorageContract):
    @pytest.fixture
    def storage(self):
        return MyStorage(...)

    @pytest.fixture
    def base_path(self, tmp_path):
        return str(tmp_path)

AuthProviderContract, JobQueueContract, and InMemoryJobQueue belong to the provisional authentication and job-queue contracts, ConnectionContract to the provisional named-connection contract, and PublishedDatasetContract to the provisional published-dataset contract, and check_registry and assert_valid_registry, which check a registry of models and genomes, to the provisional registry format. They are listed in __provisional__ and are expected to change with those contracts. FakeBigQueryClient, a recording stand-in for a BigQuery client, is provisional as well, and so are the test doubles InProcessExecutionBackend, which runs each task by calling a Python function instead of a container, and InMemoryDetailStore. InterpretationStoreContract and InterpretableModelContract check the provisional interpretation store and interpretable-model contracts. render_bucket_exports renders every result-bucket export layout through a storage, and compare_wide_tsvs (with WideTsvComparison and CellDifference) compares two wide downloads row by row, keyed by variant, and compare_prioritized_ndjson (with NdjsonComparison and FieldDifference) two prioritized NDJSON exports record by record; all belong to the provisional result-bucket export.

CONTRACT_ARTIFACT module-attribute

CONTRACT_ARTIFACT = ArtifactSchema(
    name="profile_matrix",
    version="1.0.0",
    description="Synthetic backend-neutral per-batch matrix used to verify artifact stores.",
    media_type="application/x-npz",
)

The artifact ArtifactStoreContract declares for ContractModelPlugin's run identity.

CONTRACT_VARIANT_ARTIFACT module-attribute

CONTRACT_VARIANT_ARTIFACT = ArtifactSchema(
    name="variant_tracks",
    version="1.0.0",
    description="Synthetic backend-neutral per-variant track file used to verify artifact stores.",
    media_type="application/x-npz",
    scope=ArtifactScope.VARIANT,
)

The variant-scoped artifact ArtifactStoreContract declares: one file per variant of a task's batch.

CONTRACT_MODEL_ANNOTATIONS module-attribute

CONTRACT_MODEL_ANNOTATIONS = AnnotationContract(
    source_id="org.kundajelab.altar.testing.annotations",
    schema_version="1.0.0",
    columns=(
        AnnotationColumn(
            name="region_class",
            dtype="str",
            label="Region class",
        ),
    ),
)

The synthetic annotation contract ContractModelPlugin's prioritization depends on (region_class).

CONTRACT_MODEL_GENOME_BUILD module-attribute

CONTRACT_MODEL_GENOME_BUILD = 'synthetic-v1'

The synthetic assembly contract_model_identity scores on; a conformance source declares it to be joined.

AllelicRecordBackendContract

Reusable checks for a source-agnostic allelic record backend.

Subclasses provide backend, a known query, and expected_record_count. The query should project at least two fields and may match multiple records for one key.

InMemoryAllelicRecordBackend

InMemoryAllelicRecordBackend(
    rows: Sequence[InMemoryAllelicRow],
)

A projection-aware backend double shared by annotation binding tests.

InMemoryAllelicRow dataclass

InMemoryAllelicRow(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
    fields: Mapping[str, object],
)

One physical allelic key plus opaque source-native fields.

AnnotationSourceContract

Shared checks that every AnnotationSource implementation must pass.

To use it, subclass this and provide two fixtures:

  • source — a ready-to-use AnnotationSource instance. A source that holds a connection is yielded and closed after the test; the suite checks that close can be called twice.
  • known_variant_ids — variant IDs the source has annotations for.

Override the optional contract fixture to return the AnnotationContract the source implements; the suite then checks it with assert_implements_contract.

contract

contract() -> AnnotationContract | None

Return the AnnotationContract the source implements, or None (the default) when it declares none.

test_columns_are_annotation_columns

test_columns_are_annotation_columns(source) -> None

Column declarations have unique safe names, logical types, and labels.

test_identity_is_valid_or_none

test_identity_is_valid_or_none(
    source: AnnotationSource,
) -> None

identity() returns None or a valid AnnotationSourceIdentity that survives a round trip.

test_identity_build_is_one_the_class_supports

test_identity_build_is_one_the_class_supports(
    source: AnnotationSource,
) -> None

The assembly identity() names is among the class's supported_genome_builds, when the class declares them.

test_option_descriptions_name_its_options

test_option_descriptions_name_its_options(
    source: AnnotationSource,
) -> None

Each key of the class's option_descriptions names a keyword parameter of its from_options.

test_implements_declared_contract

test_implements_declared_contract(
    source: AnnotationSource,
    contract: AnnotationContract | None,
) -> None

When the contract fixture names a contract, the source implements it.

test_implements_its_class_contract

test_implements_its_class_contract(
    source: AnnotationSource,
) -> None

When the source's class declares annotation_contract(), the instance implements that contract.

test_prioritize_predicate_is_predicate_or_none

test_prioritize_predicate_is_predicate_or_none(
    source,
) -> None

prioritize_predicate() returns a Predicate or None.

test_annotate_is_keyed_by_variant_id async

test_annotate_is_keyed_by_variant_id(
    source, known_variant_ids
) -> None

annotate() keys its result by variant ID, with a dict value for each variant.

test_annotate_omits_unknown_variant async

test_annotate_omits_unknown_variant(
    source, unknown_variant_id
) -> None

A variant the source has no row for is left out of the annotate() result.

test_output_feeds_merge_annotations async

test_output_feeds_merge_annotations(
    source, known_variant_ids
) -> None

annotate() output works as an input to merge_annotations.

InMemoryAnnotationSource

InMemoryAnnotationSource(
    name: str,
    columns: list[AnnotationColumn],
    rows: Mapping[str, Mapping[str, Any]],
    prioritize_predicate: Predicate | None = None,
    *,
    identity: AnnotationSourceIdentity | None = None,
)

Bases: AnnotationSource

An AnnotationSource backed by an in-memory dict, for use in tests.

rows maps a variant ID to its {column: value} annotations. The source is read-only, like the real reference-data sources it stands in for, so it keeps the base class's read-only add_annotations. identity is what identity() returns; pass one to stand in for a source that implements a contract.

ArtifactStoreContract

Check provenance, content addressing, coverage, lineages, idempotency, filtering, and schema collisions.

Subclass this and provide a fresh store fixture. A store that holds a connection is yielded and closed after the test; the suite checks that close can be called twice. Source files are written to tmp_path and offered through LocalStorage, as the scoring engine offers a collected output; the store copies them wherever it keeps bytes.

lineage_id staticmethod

lineage_id(identity: PluginRunIdentity) -> str

Return the score lineage a run of identity writes under, as its primary rows carry it.

source staticmethod

source(
    tmp_path: Path, name: str, content: bytes
) -> tuple[str, str]

Write one artifact file and return its URI and digest, as the engine hands them to a store.

test_lineages_coexist_and_reads_accept_named_lineages async

test_lineages_coexist_and_reads_accept_named_lineages(
    store: ArtifactStore,
    schema: ArtifactSchema,
    identity: PluginRunIdentity,
    tmp_path: Path,
) -> None

Records of two lineages coexist under one model; a read counts only the lineages it accepts.

test_accepted_lineage_with_another_schema_version_does_not_supply_files async

test_accepted_lineage_with_another_schema_version_does_not_supply_files(
    store: ArtifactStore,
    schema: ArtifactSchema,
    identity: PluginRunIdentity,
    tmp_path: Path,
) -> None

An accepted lineage's artifact stands in only when it is the same artifact at the same version.

test_variant_scoped_files_each_cover_and_list_one_variant async

test_variant_scoped_files_each_cover_and_list_one_variant(
    store: ArtifactStore,
    identity: PluginRunIdentity,
    tmp_path: Path,
) -> None

A task's per-variant files are one record each, keyed and listed by their variant.

AuthProviderContract

Checks that an AuthProvider implementation behaves as expected.

Subclass this and provide a provider fixture. If verification depends on the token contents, also override the valid_token fixture to return a token the provider accepts. The default token is an opaque string that no-auth providers ignore.

FakeBigQueryClient

FakeBigQueryClient()

Records calls and returns programmed score and annotation responses.

ConnectionContract

Shared checks that every Connection type must pass.

Subclass it and provide connection_type. Override the optional fixtures to describe the type:

  • connection_options — run spec options that build a working connection (default {}).
  • connection — the connection under test; by default connection_type.from_options(**connection_options). Override it to install fakes for a provider SDK first.
  • served_uri — a URI the connection reaches object storage for, or None if it reaches none.
  • unsupported_uri — a URI whose scheme it must decline, or None if it serves every scheme.
  • reaches_bigquery — whether bigquery_client() returns a client (default False).
  • credential_options — spec options that carry a credential, which from_options must refuse.
  • connection_holding_secret — a connection a host built in code with a secret, and the secret, or None.

credential_options

credential_options() -> dict[str, Any]

Spec options that carry a credential; by default fsspec-style keys a provider reads as a secret.

test_a_connection_reaches_a_published_copy_without_changing_its_identity

test_a_connection_reaches_a_published_copy_without_changing_its_identity(
    connection: Connection,
    served_uri: str | None,
    reaches_bigquery: bool,
) -> None

PublishedDataset.select_copy reaches each copy through this connection and leaves the identity alone.

Every binding that publishes its data resolves its run spec options with select_copy, so this is the path a connection takes into a source; the bindings' PublishedDatasetContract then checks, by reading, that a source built through a connection declares the same identity as one built without.

ContractModelPlugin

ContractModelPlugin()

Bases: ModelPlugin

Synthetic results plugin used only to verify portable Altar contracts.

DetailStoreContract

Check provenance, logical keys, parent presence, lineages, idempotency, filtering, and schema collisions.

Subclass this and provide a fresh store fixture. A store that holds a connection is yielded and closed after the test; the suite checks that close can be called twice.

lineage_id staticmethod

lineage_id(identity: PluginRunIdentity) -> str

Return the score lineage a run of identity writes under, as its primary rows carry it.

test_lineages_coexist_and_reads_accept_named_lineages async

test_lineages_coexist_and_reads_accept_named_lineages(
    store: DetailStore,
    schema: DetailSchema,
    identity: PluginRunIdentity,
) -> None

Rows of two lineages coexist under one model; a read counts only the lineages it accepts.

test_reads_take_each_variants_rows_from_the_first_accepted_lineage_with_rows async

test_reads_take_each_variants_rows_from_the_first_accepted_lineage_with_rows(
    store: DetailStore,
    schema: DetailSchema,
    identity: PluginRunIdentity,
) -> None

With several accepted lineages, each variant's rows come from one lineage, as primary reads choose one.

It is the first lineage, in the read's preference order, that holds rows for the variant. Rows of two producers never mix, even when they have different logical keys.

test_rejects_non_finite_values async

test_rejects_non_finite_values(
    store: DetailStore,
    schema: DetailSchema,
    identity: PluginRunIdentity,
) -> None

A NaN or infinite value is refused on write, and the call stores none of its rows.

test_accepted_lineage_with_another_schema_version_does_not_supply_rows async

test_accepted_lineage_with_another_schema_version_does_not_supply_rows(
    store: DetailStore,
    schema: DetailSchema,
    identity: PluginRunIdentity,
) -> None

An accepted lineage's table stands in only when it is the same table at the same version.

InMemoryDetailStore

InMemoryDetailStore()

Bases: DetailStore

A DetailStore that keeps every table in a dict, for tests of bindings and drivers.

It behaves as SqliteDetailStore does: tables are keyed by model, detail name and lineage, rows by their logical key, and a second write of a key is skipped. Rows are stored as their canonical JSON, so a read returns copies with the same values a SQLite store returns, and a NaN or infinite value is refused before any row of the call is stored.

ExecutionBackendContract

Checks that an ExecutionBackend implementation behaves as expected.

Subclass this, provide a backend fixture that returns an instance ready to submit and poll, and implement finish_task, expire_task, and launched_timeout against the fake runtime the backend is wired to. A backend whose input_digest_verification is "task" also implements fail_input_verification. A backend whose failure_detail reads task logs implements write_task_log. A backend that declares adopts_active_tasks or clears_outputs_on_fresh_start implements write_task_output and task_output, and start_task when its tasks clear their own outputs as they start. Override the spec fixture if the default trivial task cannot run on your backend, and set unlimited_timeout_s if your platform cannot run a task without a time limit.

finish_task

finish_task(
    backend: ExecutionBackend,
    handle: TaskHandle,
    *,
    exit_code: int,
) -> None

Script the fake runtime so the task behind handle has exited with exit_code.

expire_task

expire_task(
    backend: ExecutionBackend, handle: TaskHandle
) -> None

Script the fake runtime so the task behind handle has run past its timeout_s, as the platform would show it: for example a failed Job with a deadline condition, a timeout exit code, or a container that is still running long after it started.

launched_timeout

launched_timeout(
    backend: ExecutionBackend, handle: TaskHandle
) -> int | None

Return the effective deadline, in seconds, the platform would enforce for the task behind handle, or None if it would enforce none. Report what the platform applies, not only what the fake received: if the provider SDK substitutes a default when no deadline is passed, return that default.

write_task_log

write_task_log(
    backend: ExecutionBackend,
    handle: TaskHandle,
    lines: list[str],
) -> bool

Script the fake runtime so the task behind handle has written lines to its log, and return True.

The default returns False, for a backend whose failure_detail reads no log.

fail_input_verification

fail_input_verification(
    backend: ExecutionBackend, handle: TaskHandle
) -> None

Script the fake runtime so the task behind handle failed its in-task input digest check. Only a backend whose input_digest_verification is "task" needs this.

write_task_output

write_task_output(
    backend: ExecutionBackend,
    transfer: Transfer,
    content: bytes,
) -> None

Write content where a task of backend writes the output transfer, as a task would.

Only a backend that declares adopts_active_tasks or clears_outputs_on_fresh_start needs this. Provisional: the hook is expected to change.

task_output

task_output(
    backend: ExecutionBackend, transfer: Transfer
) -> bytes | None

Return the bytes at the place a task of backend writes the output transfer, or None if none.

Only a backend that declares adopts_active_tasks or clears_outputs_on_fresh_start needs this. Provisional: the hook is expected to change.

start_task

start_task(
    backend: ExecutionBackend, handle: TaskHandle
) -> None

Script the fake runtime so the task behind handle has started.

It runs any step the platform runs before the task's command, such as an init container that clears its outputs.

The default does nothing, for a backend whose submit clears outputs before it launches the task. Provisional: the hook is expected to change.

test_resubmitting_an_active_spec_does_not_run_it_twice_at_once async

test_resubmitting_an_active_spec_does_not_run_it_twice_at_once(
    backend: ExecutionBackend, spec: ContainerTaskSpec
) -> None

A driver that restarts mid-run submits a spec again while its first task is still active. The second submit must not raise. If it returns the first task (the same external_id), that handle must report the first task's outcome without being driven itself, so the work is not run twice at once. A backend that cannot recognize the earlier task launches a separate one, whose state must be its own.

test_resubmitting_after_a_terminal_outcome_runs_the_task_again async

test_resubmitting_after_a_terminal_outcome_runs_the_task_again(
    backend: ExecutionBackend,
    spec: ContainerTaskSpec,
    exit_code: int,
) -> None

Once a task has finished, succeeded or failed, submitting its spec again runs it again. The new handle must not report the old outcome: a spec names its inputs by path, and a later run may have put new content there.

test_input_digest_on_a_directory_is_rejected async

test_input_digest_on_a_directory_is_rejected(
    backend: ExecutionBackend,
    spec: ContainerTaskSpec,
    tmp_path: Path,
) -> None

A digest covers the bytes of one regular file, so a digest-bearing directory is rejected exactly like a mismatch rather than hashed by some backend-specific tree rule.

test_an_adopted_task_keeps_the_outputs_it_wrote async

test_an_adopted_task_keeps_the_outputs_it_wrote(
    backend: ExecutionBackend,
    spec: ContainerTaskSpec,
    tmp_path: Path,
) -> None

A driver that restarts submits a running task's spec again. A backend that adopts that task must keep the outputs it already wrote, or the task finishes without them.

CellDifference dataclass

CellDifference(
    variant_id: str, column: str, left: str, right: str
)

One cell that differs between two wide TSVs: its row's variant_id, its column, and both values.

FieldDifference dataclass

FieldDifference(
    variant_id: str, path: str, left: object, right: object
)

One field that differs between two prioritized NDJSON records (provisional).

path names it from the record's top: am_class, or model_scores[m1].logfc for a model's entry.

NdjsonComparison dataclass

NdjsonComparison(
    only_left: tuple[str, ...] = (),
    only_right: tuple[str, ...] = (),
    differences: tuple[FieldDifference, ...] = (),
)

The result of compare_prioritized_ndjson (provisional).

only_left and only_right are the variants one side has and the other lacks, and differences every field whose values differ. matches holds when both sides hold the same variants with equal values.

WideTsvComparison dataclass

WideTsvComparison(
    same_header: bool,
    only_left: tuple[str, ...] = (),
    only_right: tuple[str, ...] = (),
    differences: tuple[CellDifference, ...] = (),
    allowed: tuple[CellDifference, ...] = (),
)

The result of compare_wide_tsvs.

same_header says the headers are identical. only_left and only_right are the variant_ids one file has and the other lacks. differences are the differing cells outside allowed_columns, and allowed the differing cells inside them. matches holds when the headers match, both files hold the same variants, and no cell outside the allowed columns differs.

InProcessExecutionBackend

InProcessExecutionBackend(
    runtime: Callable[[ContainerTaskSpec], int | None]
    | None,
    *,
    data_root: str | PathLike[str],
    storage: Storage | None = None,
)

Bases: ExecutionBackend

Runs each task of a scoring plan by calling a Python function in place of the model's container.

The function stands in for the model runtime. It takes the task's ContainerTaskSpec, reads the staged inputs the command names, writes the outputs the task declares, and returns the exit code (None means 0). Build the ScoringRequest with this backend's path_resolver(), so the command names real files under data_root. A runtime that parses its command the way the model's CLI does is a close stand-in for the image.

submit calls the function and records its outcome; the next poll reports it. What the function prints, and the traceback of an exception it raises, become the task's log, which failure_detail returns. An exception or SystemExit fails the task with the matching exit code (1 for an exception).

Without a runtime (runtime=None), a submitted task runs until a test finishes it with finish or expire. Use that to test a driver's handling of slow, failed or timed-out tasks.

Every submit starts a fresh task. It first deletes the file at each declared output under data_root, as the local backend does, so a task never hands back a file an earlier task left there.

stage_inputs copies each input into data_root at its logical path and checks its digest there, as the local backend does. collect_outputs copies each output from data_root to its URI. A plan names the variant file and result files by their logical path, so pass on_data_root as run_scoring's input_transfer and output_transfer, and write the variant file at host_path(...).

The backend keeps what it was asked to do for a test to check: submitted (every spec, in order), staged (every staged input) and cancelled (the external IDs of cancelled tasks).

Parameters:

Name Type Description Default
runtime Callable[[ContainerTaskSpec], int | None] | None

The function that runs a task, or None to finish tasks from the test.

required
data_root str | PathLike[str]

The directory that stands in for the volume every task reads and writes.

required
storage Storage | None

The Storage that reads input URIs and writes output URIs. Defaults to LocalStorage.

None

path_resolver

path_resolver() -> PathResolver

Return the resolver that maps a logical path to its file under data_root.

host_path

host_path(logical_path: str) -> Path

Return the file under data_root that a task reads or writes at logical_path.

on_data_root

on_data_root(transfer: Transfer) -> Transfer

Point a transfer that names a layout file at its file under data_root; return any other unchanged.

A plan gives the variant file and the result files their logical path as URI, while a resource keeps the URI it was configured with. Pass this as run_scoring's input_transfer and output_transfer: the engine then reads the variant file where you wrote it, and the results where the runtime wrote them.

writes_output_in_place

writes_output_in_place(transfer: Transfer) -> bool

Whether transfer.uri is the file under data_root the task writes, rather than where collect_outputs copies it.

stage_inputs async

stage_inputs(transfers: list[Transfer]) -> None

Copy each input to its logical path under data_root and check its digest there.

A copy that does not match its digest is deleted before TransferIntegrityError is raised. An input already at its place under data_root is checked where it is and never deleted.

collect_outputs async

collect_outputs(transfers: list[Transfer]) -> None

Copy each output from its logical path under data_root to its URI.

An output the task did not write is skipped, so the scoring engine reports it as not produced.

submit async

submit(spec: ContainerTaskSpec) -> TaskHandle

Record the task and, when the backend has a runtime, run it now.

The file at each declared output is deleted first, and its directory created, so the runtime writes its files directly and a file an earlier task left there is never collected. A directory at an output's path is left alone.

poll async

poll(handles: list[TaskHandle]) -> list[TaskStatus]

Return each task's recorded state. A task this backend never ran is missing.

cancel async

cancel(handle: TaskHandle) -> None

Record the cancellation. A task that has not finished then reports reason cancelled.

failure_detail async

failure_detail(handle: TaskHandle) -> str | None

Return the last 20 lines of the task's log, or None when it wrote none or is unknown.

spec

spec(handle: TaskHandle) -> ContainerTaskSpec

Return the spec the task behind handle was submitted with.

finish

finish(handle: TaskHandle, *, exit_code: int = 0) -> None

Mark the task behind handle as exited with exit_code.

expire

expire(handle: TaskHandle) -> None

Mark the task behind handle as having run past its timeout_s.

The backend records each spec's timeout_s but does not enforce it, since a runtime call cannot be interrupted.

write_log

write_log(handle: TaskHandle, lines: Iterable[str]) -> None

Append lines to the log of the task behind handle.

InterpretableModelContract

Check a binding's interpretation methods, descriptors, workflows and rows.

Subclass this and provide the fixtures: plugin; request, an interpretation request for a fully configured model; selection, the method and target whose chain covers every method; golden_ids, the pinned interpretation ID of each method of that chain; and variants, canonical keys whose windows lie inside the reference.

InterpretationStoreContract

Check declaring, putting, committing and reading interpretation products.

Subclass this and provide a fresh store fixture. Source files are written to tmp_path and offered through LocalStorage, as ingest offers a collected output.

declare async staticmethod

declare(
    store: InterpretationStore,
    operation: InterpretationOperation,
) -> None

Record operation and declare its products: contributions and profiles upstream, hits downstream.

put async staticmethod

put(
    store: InterpretationStore,
    tmp_path: Path,
    descriptor: InterpretationDescriptor,
    product: str,
    run_id: str,
    *,
    variants: Sequence[str] = CONTRACT_VARIANTS,
    shard: tuple[tuple[str, str], ...] = (),
    content: bytes | None = None,
    model_id: str = "m1",
) -> InterpretationRecord

Write one product file and put it under descriptor's ID in execution run_id.

put_upstream async

put_upstream(
    store: InterpretationStore,
    tmp_path: Path,
    operation: InterpretationOperation,
    run_id: str,
) -> list[InterpretationRecord]

Put the upstream method's three files: contributions for each allele and one profile file.

commit_chain async

commit_chain(
    store: InterpretationStore,
    tmp_path: Path,
    operation: InterpretationOperation,
    run_id: str,
) -> tuple[
    tuple[InterpretationRecord, ...],
    tuple[InterpretationRecord, ...],
]

Put and commit both methods of operation in execution run_id.

InMemoryJobQueue

InMemoryJobQueue(kinds: Iterable[str] = ())

Bases: JobQueue

A JobQueue that records enqueued jobs in a list, used to run the conformance suite.

It is constructed with the set of kind values it accepts. enqueue records the (kind, parameters) call and returns an opaque run-<n> handle; a kind it was not told about raises KeyError, satisfying the ABC's "raise on a kind it does not recognize" contract. It is not registered as an entry-point adapter.

register_kind

register_kind(kind: str) -> None

Add a kind to the set this queue accepts.

JobQueueContract

Shared test suite that every JobQueue adapter must pass.

To run it against your adapter, subclass this and add a job_queue fixture (a fresh adapter) and a valid_kind fixture (a kind the adapter accepts). Set unknown_kind_error if the adapter raises a type more specific than Exception for an unrecognized kind.

test_enqueue_returns_opaque_handle async

test_enqueue_returns_opaque_handle(
    job_queue: JobQueue, valid_kind: str
) -> None

enqueue returns a non-empty opaque string handle for an accepted kind, without blocking on the result.

test_unknown_kind_raises async

test_unknown_kind_raises(job_queue: JobQueue) -> None

An unrecognized kind raises rather than silently succeeding (per the ABC contract).

ModelPluginContract

Bases: ModelResultContract

Backward-compatible name for the result/schema-only contract.

Compute capability checks intentionally live in InlineScorerContract and ContainerScorerContract; this class is not an all-in-one model contract.

ModelResultContract

Checks the result/schema capability of a ModelPlugin.

Subclass this and provide a plugin fixture that returns the plugin under test.

test_variant_eligibility_is_consistent

test_variant_eligibility_is_consistent(
    plugin: ModelPlugin,
) -> None

Check that the declared eligibility is well-formed and serializes, not that it matches the runtime.

This cannot catch a wrong declaration, such as an SNV-only runtime whose manifest admits indels. A binding guards that with its own test of the classes its runtime accepts.

PluginDiscoveryContract

Check one external plugin's metadata-only listing and targeted validated load.

PublishedDatasetContract

Shared checks for a binding that publishes its data and reads it in three modes.

Provide these fixtures:

  • source_class — the binding's AnnotationSource subclass, registered under its name, whose published_dataset() returns the published release and whose from_options reads the three modes.
  • built_dataset — the local directory of a small dataset the binding's builder wrote.
  • built_release_fields — the footer fields the binding declares for that dataset's release, computed as its published dataset's release_fields are.
  • known_variant_ids — canonical variant IDs the built dataset has rows for.

Optional: published (default source_class.published_dataset()) and reads_bigquery_tables (default True). A binding with no table mode returns False; its from_options then refuses table and release with a ValueError that names the option.

owned_source

owned_source(
    source_class: type[AnnotationSource],
    built_dataset: str | PathLike[str],
) -> AnnotationSource

Return the source over the built dataset, read as an owned Parquet copy.

stand_in

stand_in(
    published: PublishedDataset,
    built_dataset: str | PathLike[str],
    built_release_fields: dict[str, str],
    owned_source: AnnotationSource,
) -> Iterator[PublishedDataset]

Publish the built dataset at in-memory locations named like the real ones; remove it afterwards.

published_stand_in

published_stand_in(
    source_class: type[AnnotationSource],
    stand_in: PublishedDataset,
    monkeypatch: MonkeyPatch,
) -> PublishedDataset

Point the source's published_dataset() at the stand-in, as if its publisher had published the build.

InMemoryScoreStore

InMemoryScoreStore()

Bases: ScoreStore

A ScoreStore that keeps everything in a dict, used to run the conformance suite and the composition tests. It implements the persistence methods (including read_score) and inherits materialize from the base class.

Each row is a Mapping[str, Any] with variant_id plus the plugin's score columns. Scores are keyed by (model_id, variant_id). Re-inserting the same key does nothing, and add_scores counts it under InsertResult.skipped.

read_score async

read_score(
    model_id: str, variant_id: str
) -> dict[str, Any] | None

Return one model's stored score for a variant, or None if there isn't one.

This store implements only read_score and inherits the base class's read_scores, which reads each score by calling this method in a loop. The conformance suite therefore covers that default read_scores, while SqliteScoreStore covers a bulk override.

ScoreStoreContract

Shared test suite that every ScoreStore implementation must pass.

To run it against your store, subclass this and add a store fixture that returns a fresh, empty store instance. A store that holds a connection is yielded and closed after the test. Each test method below checks one behavior the store must have, including that close can be called twice.

creates_descriptor_table_on_write class-attribute

creates_descriptor_table_on_write: bool = True

Whether the lineage store creates its descriptor sidecar when it first records a descriptor (SQLite does), rather than requiring it to be provisioned (BigQuery).

make_annotation_source

make_annotation_source() -> Callable[..., AnnotationSource]

Return a factory for the annotation sources the materialize tests join.

The factory takes name, columns, rows ({variant_id: {column: value}}), an optional prioritize_predicate, and identity, the AnnotationSourceIdentity the source declares; these are the arguments of InMemoryAnnotationSource. The suite always passes an identity on the contract model's assembly, CONTRACT_MODEL_GENOME_BUILD, unless a test is about another assembly, since a source that declares none serves only hg38. Override it when the store can only join sources that live on its own backend.

contract_source

contract_source(
    make_annotation_source: Callable[..., AnnotationSource],
) -> Callable[..., AnnotationSource]

Return make_annotation_source with each source declaring that it serves the contract model's assembly.

Each source declares a contract of its own, named after it, so a model's dependency still resolves its columns by name.

lineage_store

lineage_store() -> Never

Return a fresh, empty store in compact-lineage mode; override it when the store has one.

descriptor_table_exists

descriptor_table_exists() -> Callable[[ScoreStore], bool]

Return a probe for whether a lineage_store's descriptor sidecar exists; override it with the mode.

provision_descriptor_table

provision_descriptor_table() -> Callable[
    [ScoreStore], Awaitable[None]
]

Return a coroutine function that provisions a lineage_store's descriptor sidecar as a host would, with one extra nullable column; override it with the mode.

test_ensure_schema_accepts_plugin_columns async

test_ensure_schema_accepts_plugin_columns(
    store, columns
) -> None

ensure_schema accepts the plugin's columns without raising.

test_add_scores_reports_rows_added async

test_add_scores_reports_rows_added(store, columns) -> None

add_scores returns an InsertResult reporting how many rows it appended.

test_primary_write_unit_commits_every_batch async

test_primary_write_unit_commits_every_batch(
    store: ScoreStore, columns
) -> None

A cleanly exited primary_write unit persists every batch its writer accepted, with per-batch counts.

test_primary_write_unit_aborts_on_error async

test_primary_write_unit_aborts_on_error(
    store: ScoreStore, columns
) -> None

An exception inside the unit propagates; a store that claims atomicity then holds none of its rows.

A pass-through store (atomic_primary_write = False) keeps the batches it already added. Either way the store accepts a new unit afterwards.

test_lineage_primary_write_unit_commits_or_aborts_as_a_whole async

test_lineage_primary_write_unit_commits_or_aborts_as_a_whole(
    lineage_store: ScoreStore, columns
) -> None

In compact-lineage mode an atomic unit's abort also leaves no lineage provenance behind.

test_get_unscored_excludes_scored_variants async

test_get_unscored_excludes_scored_variants(
    store, columns
) -> None

get_unscored_variants returns only the variants not yet scored for the model.

test_non_finite_scores_are_refused_on_write async

test_non_finite_scores_are_refused_on_write(
    store, columns, bad: float
) -> None

A NaN or infinite score raises ValueError from add_scores and from a write unit, and nothing is stored.

A missing score is None; NaN would read back as NULL from one store and as NaN from another.

test_nullable_scores_round_trip_as_null async

test_nullable_scores_round_trip_as_null(
    store, contract_source
) -> None

A None in every nullable score column reads back as None, never as NaN, False, or an empty string.

test_duplicate_writes_read_back_as_one_whole_row async

test_duplicate_writes_read_back_as_one_whole_row(
    store, contract_source
) -> None

Two writes of one (model_id, variant_id) read back as one of the two rows, whole.

A store may skip the second write or keep both rows. A read must never mix two rows' columns, such as one row's effect with the other's quality_rank. Each row leaves the other's values null, so a store that picks each column's first non-null value would mix them.

test_materialize_source_predicate_matches_quotes_backslashes_and_newlines async

test_materialize_source_predicate_matches_quotes_backslashes_and_newlines(
    store, contract_source
) -> None

A prioritizing source's string literals match values holding a quote, a backslash, or a newline.

A store that runs the predicate in SQL must escape each literal for its own dialect; GoogleSQL, for one, reads a doubled quote as a syntax error and a backslash as an escape.

ContainerScorerContract

Check labels, resources, transfers, outputs, identity, and serialization.

test_scoring_request_round_trip_preserves_the_layout

test_scoring_request_round_trip_preserves_the_layout(
    plugin: ModelPlugin, scoring_request: ScoringRequest
) -> None

A durable request must rebuild the same layout class and field values the plan was built from.

The envelope records the layout's fields and the plugin names the class, so a binding layout subclass does not revert to the model-neutral default after a restart.

InlineScorerContract

Check a model's declared in-process scoring capability and returned rows.

test_with_run_variants_carries_exactly_those_variants async

test_with_run_variants_carries_exactly_those_variants(
    plugin: ModelPlugin,
    scoring_request: ScoringRequest,
    inline_runtime: RuntimeContext | None,
) -> None

Check the hook prepare_scoring_batches binds each dense batch through.

It must change only the run inputs, leave the scientific identity alone, and score exactly the variants it was given, in order.

InMemorySqlAnnotationSource

InMemorySqlAnnotationSource(
    name: str,
    columns: list[AnnotationColumn],
    rows: Mapping[str, Mapping[str, Any]],
    prioritize_predicate: Predicate | None = None,
)

Bases: InMemoryAnnotationSource, SqlAnnotationSource

An in-memory AnnotationSource that also advertises the SQL-fusion capability, for tests.

It reuses InMemoryAnnotationSource's dict backing for annotate() and adds a sql_join that returns a synthetic JoinSpec projecting exactly its declared columns. The relation is a placeholder table reference — nothing executes it — so the double exercises the contract's shape invariants without a real SQL backend.

SqlAnnotationSourceContract

Bases: AnnotationSourceContract

Shared checks a SqlAnnotationSource must pass, on top of the base AnnotationSource contract.

Subclass this and provide the base suite's source (a SqlAnnotationSource instance) and known_variant_ids fixtures. The sql_join_context fixture below can be overridden if a source needs a non-default context.

test_source_is_sql_annotation_source

test_source_is_sql_annotation_source(source) -> None

The source advertises the capability by subclassing SqlAnnotationSource (how a store detects it).

test_sql_join_returns_joinspec_or_none

test_sql_join_returns_joinspec_or_none(
    source, sql_join_context
) -> None

sql_join returns a JoinSpec (fuse) or None (fall back to annotate()).

test_sql_join_uses_context_join_key

test_sql_join_uses_context_join_key(
    source, sql_join_context
) -> None

A returned JoinSpec joins on the context's join_key. The store builds ON <alias>.<spec.join_key> = <base>.<spec.join_key>, so a spec that picked a different key would reference a column the base relation does not expose.

test_sql_join_projects_only_declared_columns

test_sql_join_projects_only_declared_columns(
    source, sql_join_context
) -> None

A returned JoinSpec has a non-empty alias/relation and projects only columns the source declares in annotation_columns(). The store adds <alias>.<col> for each, and the materialized schema is built from annotation_columns(), so a projected column the source doesn't declare would have no schema slot.

StorageContract

Behavior every Storage implementation must satisfy.

Subclasses must provide two fixtures:

  • storage — a ready-to-use Storage instance.
  • base_path — a writable path or prefix the storage understands (a temp dir for local disk, a bucket prefix for object storage). The suite builds paths by joining onto it with /.

Subclasses may also override two optional fixtures. Each returns None by default, which skips the tests that need it:

  • list_names — a function that takes a storage directory path and returns the names of the files directly inside it. The suite uses it to check that writes leave no temporary files.
  • interrupt_writes — a function that takes a byte count and an exception and returns a context manager. Inside the context, every write and upload through the adapter stores that many bytes of its new content the way the adapter normally stores data in progress, then raises the exception. Supply it only when the adapter guarantees that a failed write leaves the previous file unchanged and creates no new file.

InMemoryVariantGeneLinkSource

InMemoryVariantGeneLinkSource(
    links: Sequence[VariantGeneLink],
    *,
    error: VariantGeneLinkSourceError | None = None,
)

Bases: VariantGeneLinkSource

Deterministic seek-paginated source used by tests and examples.

VariantGeneLinkSourceContract

Shared checks every external VariantGeneLinkSource can run.

Subclasses provide source, a known_query with at least two expected records, expected_record_ids in stable order, and an empty_query known to have no records.

VariantGeneLinkAnnotationSourceContract

Bases: AnnotationSourceContract

AnnotationSourceContract plus the projection rules of the variant-gene link annotation contract.

Subclass it with the source and known_variant_ids fixtures. The contract fixture is the link contract, so the source must declare its identity and columns, and every known variant's row must satisfy assert_link_projection.

VariantGeneLinkStoreContract

Checks visibility, idempotency, filtering, ordering, and lossless round trips.

WritableAnnotationSourceContract

Identity checks every writable annotation lake must pass; see the module docstring for the fixtures.

test_writes_are_stamped_with_the_source_identity async

test_writes_are_stamped_with_the_source_identity(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
    stored_identity_hashes: StoredHashes,
) -> None

Rows are stored with the identity, and a later source of that identity serves and counts them.

test_another_identity_sees_nothing_annotated async

test_another_identity_sees_nothing_annotated(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

A new release re-annotates: none of the old release's rows count as annotated or are served.

test_reads_do_not_mix_identities async

test_reads_do_not_mix_identities(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
    other_annotation_rows: Rows,
    stored_identity_hashes: StoredHashes,
) -> None

A lake holding two releases of one variant serves each source only its own release's values.

test_rows_carrying_another_identity_are_rejected async

test_rows_carrying_another_identity_are_rejected(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

A row stamped with another identity's hash is never written under this one.

test_legacy_rows_are_used_only_after_an_explicit_backfill async

test_legacy_rows_are_used_only_after_an_explicit_backfill(
    open_source: OpenSource,
    open_legacy_source: OpenLegacySource,
    identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

Rows written without an identity are neither served nor counted until the backfill adopts them.

example_allelic_query

example_allelic_query() -> AllelicRecordQuery

Return the small deterministic query used by the reference conformance test.

assert_implements_contract

assert_implements_contract(
    source: AnnotationSource, contract: AnnotationContract
) -> None

Assert that source declares contract and provides every contract column with the same type.

The source's identity() must name the contract's source_id, with the same major version and a version no older than the contract's. Each contract column must appear in annotation_columns() with an equal dtype, repeated flag and fields; display labels may differ. The source may declare more columns.

make_stub_sources

make_stub_sources(
    source_dir: str, plan: ContainerScoringPlan
) -> None

Create a placeholder source file for each distinct input declared by a plan.

source_transfers

source_transfers(
    transfers: list[Transfer], source_dir: str
) -> list[Transfer]

Map logical transfer paths to files under a host fixture directory.

stubify

stubify(
    spec: ContainerTaskSpec, resolver: PathResolver
) -> ContainerTaskSpec

Replace a task command with one that creates the same declared outputs.

contract_model_descriptor

contract_model_descriptor(
    runtime_identity: str = "altar-contract-runtime:v1",
    *,
    genome_build: str = CONTRACT_MODEL_GENOME_BUILD,
) -> LineageDescriptor

Return the lineage descriptor of contract_model_identity for the same runtime identity and assembly.

contract_model_identity

contract_model_identity(
    runtime_identity: str = "altar-contract-runtime:v1",
    *,
    genome_build: str = CONTRACT_MODEL_GENOME_BUILD,
) -> PluginRunIdentity

Return a complete synthetic scoring identity for persistence conformance tests.

genome_build is the assembly of the scored variants. A test that joins sources declaring no identity, which serve only hg38, passes "hg38".

compare_prioritized_ndjson

compare_prioritized_ndjson(
    left: Ndjson, right: Ndjson
) -> NdjsonComparison

Compare two prioritized NDJSON exports record by record, by value (provisional).

Each side is gzipped NDJSON, as bytes or a local path, or a sequence of parts. Records are keyed by variant_id, and each record's model_scores entries by model_id, so order never matters. Values compare as values_equal compares them: an omitted key equals null (BigQuery's JSON export omits a null field, Altar writes it), numbers compare by float64 value, an INT64 written as a JSON string equals the number, and "NaN", "Infinity" and "-Infinity" equal the floats. Every differing field is reported. A variant_id that appears twice on one side raises ValueError.

The left records are spilled to a scratch SQLite database keyed by variant_id and each right record is looked up there, so memory holds a record at a time (and the differences found).

compare_wide_tsvs

compare_wide_tsvs(
    left: WideTsv,
    right: WideTsv,
    *,
    allowed_columns: Iterable[str] = (),
    nan_equals_empty: bool = False,
    semantic: bool = False,
) -> WideTsvComparison

Compare two gzipped wide TSVs as sets of rows keyed by variant_id, cell by cell (provisional).

Each file is the bytes of a gzipped TSV or a local path to one. Row order is ignored. Every differing cell is listed, under allowed when its column is in allowed_columns (a display rider's columns, whose ties and NULL decisions may be broken differently by another export path) and under differences otherwise. Cells of a row only one file has are not compared. A variant_id that appears twice in one file raises ValueError. nan_equals_empty counts a NaN cell (NaN or nan) and an empty one as equal, for comparing with a file written by an exporter that wrote NaN as an empty cell. Altar's TSV writes NaN, and BigQuery's CSV export nan; semantic=True counts those two as equal.

semantic compares values rather than text, for comparing Altar's download with one BigQuery's own export wrote, whose text differs while its values are equal (cells_equal): a number by its float64 value after parsing, so 0.3333333333333333 (Altar, the shortest text that round-trips) equals 0.33333333333333331 (BigQuery, up to 17 digits) and 1e+15 equals 1000000000000000.0; nan, inf and -inf equal NaN, Infinity and -Infinity; and a JSON-text cell (a repeated or struct annotation, a reason list) by its parsed value under the same rules. Any other cell is compared as text.

Both files are read as streams: the left file's rows are spilled to a scratch SQLite database keyed by variant_id, and each right row is looked up there, so memory holds a row at a time (and the differences found), whatever the files' size, such as a download of 10,000 columns by 50,000 variants.

render_bucket_exports async

render_bucket_exports(
    buckets: ResultBuckets, root: str, storage: Storage
) -> dict[str, bytes]

Render every bucket export layout under root through storage, and return each file's bytes by path.

The paths, relative to root: wide/ (the wide TSV and its column manifest), prioritized/ (the prioritized wide TSV), parquet/ (the wide Parquet), ndjson/ (the prioritized NDJSON), and long-tsv/ and long-parquet/ (the long layout in each format).

assert_valid_registry

assert_valid_registry(
    uri: str, *, connection: Connection | None = None
) -> None

Raise AssertionError listing every problem check_registry finds with the registry at uri.

check_registry

check_registry(
    uri: str,
    *,
    connection: Connection | None = None,
    allow_private_hosts: bool = False,
) -> list[str]

Return every problem with the registry whose index is at uri, and with each entry it names; empty if none.

It reads the index and every entry document, and no other file. Each document must parse, each entry's bytes must have the digest its path gives, and each index line must agree with its entry. Every URI must be one a published registry may name (altar.plugins.registries.documents.uri_problem with publishing). When the entry's binding is installed, its configuration must load. An index hosted elsewhere is read as a fetch reads it, and its entries are read only where a fetch would read them (RegistrySource.check_uri).

assert_link_projection(row: Mapping[str, object]) -> None

Assert that one variant's annotation row follows the contract's projection rules.

link_genes is a non-empty list of records with exactly the LINK_GENE_FIELDS, each naming a gene and a link record, ordered strongest first with unrankable links last. link_count and link_gene_count count those records and their distinct gene IDs. link_top_gene and link_top_score are either both absent, when the first link cannot be ranked, or describe the first link; link_top_gene_symbol is present only with them.