Conformance suites¶

Extension packages should import reusable contracts and fixtures from altar.testing.

AuthProviderContract, JobQueueContract, and InMemoryJobQueue are provisional, like the contracts they exercise.

testing ¶

Stable conformance suites and fixtures for external Altar extensions.

Import a contract and subclass it to prove your adapter satisfies a protocol — the same suites core runs against its own reference adapters. Pytest is a base dependency so this stable façade is always importable; the test extra adds the remaining tools used by Altar's own suite.

from altar.testing import StorageContract

class TestMyStorage(StorageContract):
    @pytest.fixture
    def storage(self):
        return MyStorage(...)

    @pytest.fixture
    def base_path(self, tmp_path):
        return str(tmp_path)

AuthProviderContract, JobQueueContract, and InMemoryJobQueue belong to the provisional authentication and job-queue contracts. They are listed in __provisional__ and excluded from the stable-API commitment until those contracts are promoted.

CONTRACT_MODEL_ANNOTATIONS `module-attribute` ¶

CONTRACT_MODEL_ANNOTATIONS = AnnotationContract(
    source_id="org.kundajelab.altar.testing.annotations",
    schema_version="1.0.0",
    columns=(
        AnnotationColumn(
            name="region_class",
            dtype="str",
            label="Region class",
        ),
    ),
)

The synthetic annotation contract ContractModelPlugin's prioritization depends on (region_class).

AllelicRecordBackendContract ¶

Reusable checks for a source-agnostic allelic record backend.

Subclasses provide backend, a known query, and expected_record_count. The query should project at least two fields and may match multiple records for one key.

InMemoryAllelicRecordBackend ¶

InMemoryAllelicRecordBackend(
    rows: Sequence[InMemoryAllelicRow],
)

A projection-aware backend double shared by annotation binding tests.

InMemoryAllelicRow `dataclass` ¶

InMemoryAllelicRow(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
    fields: Mapping[str, object],
)

One physical allelic key plus opaque source-native fields.

AnnotationSourceContract ¶

Shared checks that every AnnotationSource implementation must pass.

To use it, subclass this and provide two fixtures:

source — a ready-to-use AnnotationSource instance.
known_variant_ids — variant IDs the source has annotations for.

Override the optional contract fixture to return the AnnotationContract the source implements; the suite then checks it with assert_implements_contract.

contract ¶

contract() -> AnnotationContract | None

Return the AnnotationContract the source implements, or None (the default) when it declares none.

test_columns_are_annotation_columns ¶

test_columns_are_annotation_columns(source) -> None

Column declarations have unique safe names, logical types, and labels.

test_identity_is_valid_or_none ¶

test_identity_is_valid_or_none(
    source: AnnotationSource,
) -> None

identity() returns None or a valid AnnotationSourceIdentity that survives a round trip.

test_implements_declared_contract ¶

test_implements_declared_contract(
    source: AnnotationSource,
    contract: AnnotationContract | None,
) -> None

When the contract fixture names a contract, the source implements it.

test_prioritize_predicate_is_predicate_or_none ¶

test_prioritize_predicate_is_predicate_or_none(
    source,
) -> None

prioritize_predicate() returns a Predicate or None.

test_annotate_is_keyed_by_variant_id `async` ¶

test_annotate_is_keyed_by_variant_id(
    source, known_variant_ids
) -> None

annotate() keys its result by variant ID, with a dict value for each variant.

test_annotate_omits_unknown_variant `async` ¶

test_annotate_omits_unknown_variant(
    source, unknown_variant_id
) -> None

A variant the source has no row for is left out of the annotate() result.

test_output_feeds_merge_annotations `async` ¶

test_output_feeds_merge_annotations(
    source, known_variant_ids
) -> None

annotate() output works as an input to merge_annotations.

InMemoryAnnotationSource ¶

InMemoryAnnotationSource(
    name: str,
    columns: list[AnnotationColumn],
    rows: Mapping[str, Mapping[str, Any]],
    prioritize_predicate: Predicate | None = None,
    *,
    identity: AnnotationSourceIdentity | None = None,
)

Bases: AnnotationSource

An AnnotationSource backed by an in-memory dict, for use in tests.

rows maps a variant ID to its {column: value} annotations. The source is read-only, like the real reference-data sources it stands in for, so it keeps the base class's read-only add_annotations. identity is what identity() returns; pass one to stand in for a source that implements a contract.

AuthProviderContract ¶

Checks that an AuthProvider implementation behaves as expected.

Subclass this and provide a provider fixture. If verification depends on the token contents, also override the valid_token fixture to return a token the provider accepts. The default token is an opaque string that no-auth providers ignore.

ContractModelPlugin ¶

ContractModelPlugin()

Bases: ModelPlugin

Synthetic results plugin used only to verify portable Altar contracts.

DetailStoreContract ¶

Check provenance, logical keys, parent presence, idempotency, filtering, and schema collisions.

ExecutionBackendContract ¶

Checks that an ExecutionBackend implementation behaves as expected.

Subclass this, provide a backend fixture that returns an instance ready to submit and poll, and implement finish_task, expire_task, and launched_timeout against the fake runtime the backend is wired to. A backend whose input_digest_verification is "task" also implements fail_input_verification. Override the spec fixture if the default trivial task cannot run on your backend, and set unlimited_timeout_s if your platform cannot run a task without a time limit.

finish_task ¶

finish_task(
    backend: ExecutionBackend,
    handle: TaskHandle,
    *,
    exit_code: int,
) -> None

Script the fake runtime so the task behind handle has exited with exit_code.

expire_task ¶

expire_task(
    backend: ExecutionBackend, handle: TaskHandle
) -> None

Script the fake runtime so the task behind handle has run past its timeout_s, as the platform would show it: for example a failed Job with a deadline condition, a timeout exit code, or a container that is still running long after it started.

launched_timeout ¶

launched_timeout(
    backend: ExecutionBackend, handle: TaskHandle
) -> int | None

Return the effective deadline, in seconds, the platform would enforce for the task behind handle, or None if it would enforce none. Report what the platform applies, not only what the fake received: if the provider SDK substitutes a default when no deadline is passed, return that default.

fail_input_verification ¶

fail_input_verification(
    backend: ExecutionBackend, handle: TaskHandle
) -> None

Script the fake runtime so the task behind handle failed its in-task input digest check. Only a backend whose input_digest_verification is "task" needs this.

test_resubmitting_an_active_spec_does_not_run_it_twice_at_once `async` ¶

test_resubmitting_an_active_spec_does_not_run_it_twice_at_once(
    backend: ExecutionBackend, spec: ContainerTaskSpec
) -> None

A driver that restarts mid-run submits a spec again while its first task is still active. The second submit must not raise. If it returns the first task (the same external_id), that handle must report the first task's outcome without being driven itself, so the work is not run twice at once. A backend that cannot recognize the earlier task launches a separate one, whose state must be its own.

test_resubmitting_after_a_terminal_outcome_runs_the_task_again `async` ¶

test_resubmitting_after_a_terminal_outcome_runs_the_task_again(
    backend: ExecutionBackend,
    spec: ContainerTaskSpec,
    exit_code: int,
) -> None

Once a task has finished, succeeded or failed, submitting its spec again runs it again. The new handle must not report the old outcome: a spec names its inputs by path, and a later run may have put new content there.

test_input_digest_on_a_directory_is_rejected `async` ¶

test_input_digest_on_a_directory_is_rejected(
    backend: ExecutionBackend,
    spec: ContainerTaskSpec,
    tmp_path: Path,
) -> None

A digest covers the bytes of one regular file, so a digest-bearing directory is rejected exactly like a mismatch rather than hashed by some backend-specific tree rule.

InMemoryJobQueue ¶

InMemoryJobQueue(kinds: Iterable[str] = ())

Bases: JobQueue

A JobQueue that records enqueued jobs in a list, used to run the conformance suite.

It is constructed with the set of kind values it accepts. enqueue records the (kind, parameters) call and returns an opaque run-<n> handle; a kind it was not told about raises KeyError, satisfying the ABC's "raise on a kind it does not recognize" contract. It is not registered as an entry-point adapter.

register_kind ¶

register_kind(kind: str) -> None

Add a kind to the set this queue accepts.

JobQueueContract ¶

Shared test suite that every JobQueue adapter must pass.

To run it against your adapter, subclass this and add a job_queue fixture (a fresh adapter) and a valid_kind fixture (a kind the adapter accepts). Set unknown_kind_error if the adapter raises a type more specific than Exception for an unrecognized kind.

test_enqueue_returns_opaque_handle `async` ¶

test_enqueue_returns_opaque_handle(
    job_queue: JobQueue, valid_kind: str
) -> None

enqueue returns a non-empty opaque string handle for an accepted kind, without blocking on the result.

test_unknown_kind_raises `async` ¶

test_unknown_kind_raises(job_queue: JobQueue) -> None

An unrecognized kind raises rather than silently succeeding (per the ABC contract).

ModelPluginContract ¶

Bases: ModelResultContract

Backward-compatible name for the result/schema-only contract.

Compute capability checks intentionally live in InlineScorerContract and ContainerScorerContract; this class is not an all-in-one model contract.

ModelResultContract ¶

Checks the result/schema capability of a ModelPlugin.

Subclass this and provide a plugin fixture that returns the plugin under test.

test_variant_eligibility_is_consistent ¶

test_variant_eligibility_is_consistent(
    plugin: ModelPlugin,
) -> None

Check that the declared eligibility is well-formed and serializes, not that it matches the runtime.

This cannot catch a wrong declaration, such as an SNV-only runtime whose manifest admits indels. A binding guards that with its own test of the classes its runtime accepts.

PluginDiscoveryContract ¶

Check one external plugin's metadata-only listing and targeted validated load.

InMemoryScoreStore ¶

InMemoryScoreStore()

Bases: ScoreStore

A ScoreStore that keeps everything in a dict, used to run the conformance suite and the composition tests. It implements the persistence methods (including read_score) and inherits materialize from the base class.

Each row is a Mapping[str, Any] with variant_id plus the plugin's score columns. Scores are keyed by (model_id, variant_id). Re-inserting the same key does nothing, and add_scores counts it under InsertResult.skipped.

read_score `async` ¶

read_score(
    model_id: str, variant_id: str
) -> dict[str, Any] | None

Return one model's stored score for a variant, or None if there isn't one.

This store implements only read_score and inherits the base class's read_scores, which reads each score by calling this method in a loop. The conformance suite therefore covers that default read_scores, while SqliteScoreStore covers a bulk override.

ScoreStoreContract ¶

Shared test suite that every ScoreStore implementation must pass.

To run it against your store, subclass this and add a store fixture that returns a fresh, empty store instance. Each test method below checks one behavior the store must have.

make_annotation_source ¶

make_annotation_source() -> Callable[..., AnnotationSource]

Return a factory for the annotation sources the materialize tests join.

The factory takes name, columns, rows ({variant_id: {column: value}}), and an optional prioritize_predicate, the arguments of InMemoryAnnotationSource. Override it when the store can only join sources that live on its own backend.

lineage_store ¶

lineage_store() -> Never

Return a fresh, empty store in compact-lineage mode; override it when the store has one.

test_ensure_schema_accepts_plugin_columns `async` ¶

test_ensure_schema_accepts_plugin_columns(
    store, columns
) -> None

ensure_schema accepts the plugin's columns without raising.

test_add_scores_reports_rows_added `async` ¶

test_add_scores_reports_rows_added(store, columns) -> None

add_scores returns an InsertResult reporting how many rows it appended.

test_get_unscored_excludes_scored_variants `async` ¶

test_get_unscored_excludes_scored_variants(
    store, columns
) -> None

get_unscored_variants returns only the variants not yet scored for the model.

ContainerScorerContract ¶

Check labels, resources, transfers, outputs, identity, and serialization.

InlineScorerContract ¶

Check a model's declared in-process scoring capability and returned rows.

InMemorySqlAnnotationSource ¶

InMemorySqlAnnotationSource(
    name: str,
    columns: list[AnnotationColumn],
    rows: Mapping[str, Mapping[str, Any]],
    prioritize_predicate: Predicate | None = None,
)

Bases: InMemoryAnnotationSource, SqlAnnotationSource

An in-memory AnnotationSource that also advertises the SQL-fusion capability, for tests.

It reuses InMemoryAnnotationSource's dict backing for annotate() and adds a sql_join that returns a synthetic JoinSpec projecting exactly its declared columns. The relation is a placeholder table reference — nothing executes it — so the double exercises the contract's shape invariants without a real SQL backend.

SqlAnnotationSourceContract ¶

Bases: AnnotationSourceContract

Shared checks a SqlAnnotationSource must pass, on top of the base AnnotationSource contract.

Subclass this and provide the base suite's source (a SqlAnnotationSource instance) and known_variant_ids fixtures. The sql_join_context fixture below can be overridden if a source needs a non-default context.

test_source_is_sql_annotation_source ¶

test_source_is_sql_annotation_source(source) -> None

The source advertises the capability by subclassing SqlAnnotationSource (how a store detects it).

test_sql_join_returns_joinspec_or_none ¶

test_sql_join_returns_joinspec_or_none(
    source, sql_join_context
) -> None

sql_join returns a JoinSpec (fuse) or None (fall back to annotate()).

test_sql_join_uses_context_join_key ¶

test_sql_join_uses_context_join_key(
    source, sql_join_context
) -> None

A returned JoinSpec joins on the context's join_key. The store builds ON <alias>.<spec.join_key> = <base>.<spec.join_key>, so a spec that picked a different key would reference a column the base relation does not expose.

test_sql_join_projects_only_declared_columns ¶

test_sql_join_projects_only_declared_columns(
    source, sql_join_context
) -> None

A returned JoinSpec has a non-empty alias/relation and projects only columns the source declares in annotation_columns(). The store adds <alias>.<col> for each, and the materialized schema is built from annotation_columns(), so a projected column the source doesn't declare would have no schema slot.

StorageContract ¶

Behavior every Storage implementation must satisfy.

Subclasses must provide two fixtures:

storage — a ready-to-use Storage instance.
base_path — a writable path or prefix the storage understands (a temp dir for local disk, a bucket prefix for object storage). The suite builds paths by joining onto it with /.

InMemoryVariantGeneLinkSource ¶

InMemoryVariantGeneLinkSource(
    links: Sequence[VariantGeneLink],
    *,
    error: VariantGeneLinkSourceError | None = None,
)

Bases: VariantGeneLinkSource

Deterministic seek-paginated source used by tests and examples.

VariantGeneLinkSourceContract ¶

Shared checks every external VariantGeneLinkSource can run.

Subclasses provide source, a known_query with at least two expected records, expected_record_ids in stable order, and an empty_query known to have no records.

VariantGeneLinkStoreContract ¶

Checks visibility, idempotency, filtering, ordering, and lossless round trips.

WritableAnnotationSourceContract ¶

Identity checks every writable annotation lake must pass; see the module docstring for the fixtures.

test_writes_are_stamped_with_the_source_identity `async` ¶

test_writes_are_stamped_with_the_source_identity(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
    stored_identity_hashes: StoredHashes,
) -> None

Rows are stored with the identity, and a later source of that identity serves and counts them.

test_another_identity_sees_nothing_annotated `async` ¶

test_another_identity_sees_nothing_annotated(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

A new release re-annotates: none of the old release's rows count as annotated or are served.

test_reads_do_not_mix_identities `async` ¶

test_reads_do_not_mix_identities(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
    other_annotation_rows: Rows,
    stored_identity_hashes: StoredHashes,
) -> None

A lake holding two releases of one variant serves each source only its own release's values.

test_rows_carrying_another_identity_are_rejected `async` ¶

test_rows_carrying_another_identity_are_rejected(
    open_source: OpenSource,
    identity: AnnotationSourceIdentity,
    other_identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

A row stamped with another identity's hash is never written under this one.

test_legacy_rows_are_used_only_after_an_explicit_backfill `async` ¶

test_legacy_rows_are_used_only_after_an_explicit_backfill(
    open_source: OpenSource,
    open_legacy_source: OpenLegacySource,
    identity: AnnotationSourceIdentity,
    annotation_rows: Rows,
) -> None

Rows written without an identity are neither served nor counted until the backfill adopts them.

example_allelic_query ¶

example_allelic_query() -> AllelicRecordQuery

Return the small deterministic query used by the reference conformance test.

assert_implements_contract ¶

assert_implements_contract(
    source: AnnotationSource, contract: AnnotationContract
) -> None

Assert that source declares contract and provides every contract column with the same type.

The source's identity() must name the contract's source_id, with the same major version and a version no older than the contract's. Each contract column must appear in annotation_columns() with an equal dtype, repeated flag and fields; display labels may differ. The source may declare more columns.

make_stub_sources ¶

make_stub_sources(
    source_dir: str, plan: ContainerScoringPlan
) -> None

Create a placeholder source file for each distinct input declared by a plan.

source_transfers ¶

source_transfers(
    transfers: list[Transfer], source_dir: str
) -> list[Transfer]

Map logical transfer paths to files under a host fixture directory.

stubify ¶

stubify(
    spec: ContainerTaskSpec, resolver: PathResolver
) -> ContainerTaskSpec

Replace a task command with one that creates the same declared outputs.

contract_model_identity ¶

contract_model_identity(
    runtime_identity: str = "altar-contract-runtime:v1",
) -> PluginRunIdentity

Return a complete synthetic scoring identity for persistence conformance tests.