Maturity labels¶
Altar is in beta, at version 0.x. Every module, command, run-spec section and backend of the altar and
altar-identity packages carries one of three maturity labels. The tables below list which has which, and which
import paths are public.
Labels¶
| Label | What it means |
|---|---|
| Stable | Works and is documented. Its names and behaviour are settled. |
| Provisional | Works and is documented. Its names and behaviour are still being designed. |
| Preview | Works and is documented, but has been tested only against fakes, emulators or some real conditions. The table below says which. |
What has which label¶
| What | Label | Notes |
|---|---|---|
altar.models |
Stable | Except the provisional names listed below |
altar.run |
Stable | Except the preparation, validation, interpretation and per-source summary names below |
altar.scoring |
Stable | Except ScoringOutputError |
altar.execution |
Stable | Except the job-queue names, MissingTaskStatusError, and the GPU pool check (ExecutionBackend.gpu_pools, require_gpu_pool, UnknownPoolError) |
altar.results |
Stable | Except the BigQuery export helpers below |
altar.sources |
Stable | Except the published-dataset, builder, link-dataset and position-score names below |
altar.predicates |
Stable | Except SqlDialect |
altar.registry |
Stable | |
altar.variants |
Stable | Except the streaming-ingest schemas and the augmented-VCF names below |
altar.testing |
Stable | Except the suites and test doubles below |
altar_identity |
Stable | See Identity package |
altar.interpretation, altar interpret, altar interpretations, [interpretations.<name>] and [interpretation_store] in a run spec |
Provisional | Every name |
altar.connections, [connections] in a run spec |
Provisional | Every name |
altar.access |
Provisional | Every name |
altar.registries, altar resources, the registry format, [registries], [variants] reference and [models.<name>] resources in a run spec |
Provisional | Every name |
altar run, altar spec-schema, altar models, altar sources |
Stable | The text they print while running is for people to read. A program reads the export and the run summary |
altar run --validate, altar prepare |
Provisional | They run through provisional names of altar.run |
altar pull |
Provisional | It runs through the provisional published-dataset names of altar.sources |
| The run-spec file format | Provisional | A spec carries no format version |
[export] layout = "long" in a run spec, and the long tables it writes |
Provisional | Every table, column and file name |
_columns.tsv, the list of a wide export's columns |
Provisional | Its columns and the values of kind |
Local Docker execution backend (local) |
Stable | |
| Kubernetes and Modal execution backends | Preview | Tested with the execution-backend conformance suite against fake clients only; not run on a live cluster or a live Modal account |
| SQLite score, detail and annotation stores; SQLite and in-memory variant–gene link stores | Stable | |
Storage adapters local, https and fsspec, and the storage artifact store |
Stable | |
BigQuery score and detail stores, BigQuery sources, and the table option of a source with a published dataset |
Preview | Tested on the BigQuery emulator and on DuckDB; not run against live BigQuery |
altar pull --bigquery |
Preview | Tested against a fake BigQuery client only; not run against live BigQuery |
The Python names of the BigQuery store and sources, such as BigQueryScoreStore, are Stable, except the
provisional names listed below. Their Preview label describes how the backend was tested.
Platform support lists the test evidence for each backend, store and storage adapter.
Identity package¶
VariantKey and the other identity rules are defined in altar-identity (import altar_identity), a separate
distribution that altar depends on. The public modules re-export its names as the same objects, so an
isinstance check or an except clause behaves the same whichever path imported them:
| Names | Re-exported by |
|---|---|
VariantKey, VariantIdentityError, canonical_chromosome, canonical_variant_id |
altar.models, altar.sources, altar.variants |
has_verification_record, VERIFICATION_RECORD_SUFFIX |
altar.execution |
These re-exports carry the same label as the rest of their module. Bindings import them through a public module,
not from altar_identity. altar.variants.read_variants_frame loads a variant table as a DataFrame;
altar_identity.read_variants yields VariantKey values from a container variant file.
Model runtimes import altar_identity directly, because most of them run on a Python version that cannot install
Altar core. They use the names in its __all__. read_variants, read_variant_rows and batched read the variant
file Altar stages for a container. sha256_file, verify_file and the verification-record helpers check staged
resources.
altar-identity supports Python 3.9 and later, and its wheel declares no dependencies. altar pins it with ==,
so each Altar release requires exactly one altar-identity version, and a runtime image built from the same commit
runs the same identity code.
Provisional names¶
A provisional name is importable from a stable public module like any other name. Each module lists its
provisional names in __provisional__, a tuple that is a subset of __all__:
The API reference pages say which names are provisional. A method or attribute of a stable class can be
provisional too. Its docstring then says so, and the reference page shows that docstring. An entry-point group is
provisional when the contract it registers is: altar.auth, altar.job_queues, altar.connections and
altar.registries. See
Entry-point groups.
| Module | Provisional names |
|---|---|
altar.execution |
JobQueue, get_job_queue_registry (entry-point group altar.job_queues) |
altar.execution |
MissingTaskStatusError, raised when a backend's poll returns no status for a task |
altar.execution |
UnknownPoolError and require_gpu_pool, the check of a GPU pool name against the pools a backend defines (ExecutionBackend.gpu_pools, also provisional) |
altar.access |
Every name: AuthIdentity, AuthProvider, DevAuthProvider (for development only), EnvironmentSecretResolver, InvalidTokenError, get_auth_registry (entry-point group altar.auth) |
altar.connections |
Every name: Connection, FsspecConnection, GcpConnection, get_connection_registry (entry-point group altar.connections) |
altar.registries |
Every name, among them RegistryIndex, RegistryEntry, FileReference, PackagedRegistry and get_registry_source_registry (entry-point group altar.registries) |
altar.sources |
PublishedDataset (with its cluster_by), PublishedLocation, and the AnnotationSource.published_dataset hook that returns one; PublishedCopy, OwnedDatasetCopy and OwnedTableCopy, which PublishedDataset.select_copy returns; pull_dataset, pull_to_bigquery, PullSummary, BigQueryLoadSummary and PullProgress, behind altar pull; bigquery_release_labels and check_bigquery_release, which record and check a BigQuery copy's release as table labels; the AnnotationSource.annotation_contract hook, with which altar sources lists a class's columns without building it |
altar.sources |
BuildOutput, DEFAULT_INPUT_ATTEMPTS, storage_option_connection and add_release_argument, with which a builder writes to a URI, retries a remote input, and takes storage options and --release from its command line; canonical_chromosome_runs, minimal_representations and the methods OpenInput.header_lines, OpenInput.tsv_batches, OpenInput.convert_batches, RecordOrder.advance_batch and PartitionedWriter.append_batch, with which a builder parses, converts and writes Arrow record batches |
altar.sources |
write_link_dataset, LinkDatasetSummary, link_dataset_row, link_dataset_release_fields and LINK_DATASET_METADATA: the Parquet link dataset layout (one typed row per link, chromosome= partitions, 32,000-row groups sorted by start) and its writer; VariantGeneLinkAnnotationSource.from_dataset, which reads it, and from_bigquery_table, which reads it loaded into BigQuery |
altar.sources |
POSITION_SCORE_METADATA, position_score_release_fields and PositionScoreAnnotationSource: the Parquet position score layout (one row per scored base, chr= partitions, pos strictly increasing, 128,000-row groups) and the source that reads it, with its indels option; the parquet_options keyword of PartitionedWriter |
altar.sources |
allelic_key, which turns a canonical variant ID into the key an AllelicRecordBackend looks up, and minimal_variant_key, the trimmed key that allelic_key and the position and link sources look up |
altar.sources |
unknown_options_message and owned_dataset_storage_options, with which a source's from_options refuses an unknown option and reads an owned dataset through a named connection |
altar.predicates |
SqlDialect, which Predicate.to_sql and Term.to_sql take as dialect |
altar.run |
SourceRunSummary, the per-source provenance in RunSummary.sources and _run_summary.json |
altar.run |
ModelSectionSummary, each section whose resources lists several registry entries, in RunSummary.sections and RunPlan.sections |
altar.run |
PotentialMiss, the counts in RunSummary.potential_misses of cohort rows a lookup may miss; validate and the RunPlan it returns, behind altar run --validate |
altar.run |
prepare, validate_preparation, PreparationSummary and ModelPreparationSummary, behind altar prepare |
altar.run |
interpret, validate_interpretations, InterpretationSettings, InterpretationStoreSettings, InterpretationsSummary, InterpretationOperationSummary and InterpretationProduct, behind altar interpret and the [interpretations.<name>] and [interpretation_store] spec sections |
altar.interpretation |
Every name: InterpretableModel, InterpretationMethod, InterpretationSelection, InterpretationDescriptor, InterpretationOperation, the preparation, workflow, ingest and read types, prepare_interpretation, ingest_interpretation_results, read_interpretation, interpret, InterpretationRunResult, InterpretationRunError, InterpretationStore, StorageInterpretationStore and their key, record and row types |
altar.scoring |
ScoringOutputError, raised when a run's output holds a row for a variant outside the submitted batch, or a terminal output was not produced |
altar.results |
VariantIdentity, DisplayRider, RiderColumn, export_col_alias, safe_identifier, and the BigQueryScoreStore.count_unscored method, which an application uses to run the BigQuery store in production (Run at warehouse scale on BigQuery) |
altar.results |
wide_column_manifest, ColumnManifestEntry, column_manifest_tsv, write_column_manifest and COLUMN_MANIFEST_FILE, which list a wide table's columns; write_long_export, long_score_tables, LongScoreTable and LongModelInfo, which write the long format (Export results) |
altar.results |
ResultBuckets, ResultBucket, ResultBucketDescriptor, RESULT_BUCKET_SCHEMA_VERSION, result_bucket_schema, genomic_sort_key, bucket_export_layout, bucket_rows_for and plan_buckets, the form in which a store hands an export to the writers; render_wide_tsv, render_wide_parquet, render_prioritized_ndjson, render_long_export and render_occurrence_buckets, which write it to any storage; ExportStaging and ExportScanLimitExceeded, BigQuery's staging setting and its cost guard, and ExportDuplicateRowsError, the refusal of an export that repeats a variant (Export results) |
altar.testing |
AuthProviderContract, ConnectionContract, InMemoryJobQueue, JobQueueContract, PublishedDatasetContract, check_registry, assert_valid_registry |
altar.testing |
render_bucket_exports, compare_wide_tsvs, WideTsvComparison and CellDifference, which render every export layout and compare two wide downloads by variant, and compare_prioritized_ndjson, NdjsonComparison and FieldDifference, which compare two prioritized NDJSON exports by value |
altar.testing |
InterpretationStoreContract, InterpretableModelContract |
altar.testing |
InProcessExecutionBackend and InMemoryDetailStore, test doubles that run a model's tasks through a Python function and keep detail rows in memory |
altar.testing |
FakeBigQueryClient, the recording client Altar's own BigQuery tests use |
altar.variants |
OCCURRENCE_ARROW_SCHEMA, CANONICAL_ARROW_SCHEMA, INVALID_ARROW_SCHEMA, canonical_sample_selection_digest |
altar.variants |
AnnotationInfoField, check_annotation_info_fields and augment_run_export, with the vcf_info_fields() hook a source class declares its augmented-VCF fields with (Write an augmented VCF) |
These methods and attributes of stable classes are provisional, and their docstrings say so:
| Member | What it does |
|---|---|
ModelPlugin.reads_reference_genome |
Declares that a model reads no local reference genome, so the reference stays out of its identity |
ModelPlugin.prepared_run_inputs |
Checks the files altar prepare published against a scoring request |
ModelPlugin.run_variant_ids |
Returns the variants an inline plan scores, so the engine can check its output rows |
InterpretableModel hooks |
The methods a binding implements to declare interpretation methods |
ExecutionBackend.adopts_active_tasks, clears_outputs_on_fresh_start, writes_output_in_place |
Declare how a backend treats the outputs of a task it resubmits or adopts |
ExecutionBackendContract.write_task_output, task_output, start_task |
The hooks the backend suite uses to test output clearing |
ModalExecutionBackend.remove_from_volume |
Deletes a stale output from a Modal Volume before a task starts |
WorkflowExecutionError.completed_stages, collected_outputs, and run_model_workflow(collect_completed=...) |
Report and keep the stages of a workflow that finished before a failure |
ModelSettings.entries, ModelSettings.section and ModelRunSummary.section |
The registry entries a section's resources names, and the section whose list a model was expanded from |
ExportSettings.layout, RunSummary.column_manifest_uri and RunSummary.export_tables |
Choose the wide or long export, and name a wide export's column list and a long export's tables |
Private namespaces¶
altar.plugins.*, modules belowaltar.testing, and other implementation modules are private. They can be imported, but they are not part of the documented interface, and a package outside Altar should not depend on them.- Command modules below
altar.variantsare not public Python imports. The namesaltar.variantsitself exports are. - The package root,
altar, exports only__version__.
Every stable and provisional name imports in a plain pip install altar environment, without optional cloud SDKs,
model frameworks, credentials or a source checkout.
Typing scope¶
The altar distribution ships py.typed. Strict mypy checks the public modules, the types their exported
signatures reach, and every binding. It does not check some implementation modules below altar.variants.
Next¶
- Platform support: how each backend, store and storage adapter was tested.
- Package and register a plugin: how to version a binding, and which changes to its manifest change stored results.