Skip to content

Upgrade from 0.2 to 0.3

Altar 0.3.0 adds lineage equivalence: when a store reads scores, it also reuses the rows of older lineages that the binding declares scientifically the same. Most of the steps below are for an application that keeps scores in an Altar score store from its own code: what to provision, which calls to change, and how to backfill existing lineages.

0.3.0 also lets a model score variants on another assembly than the one it was trained on, which changes how Altar records genome builds. Variant and training assemblies covers that upgrade. The rest of 0.3.0's breaking changes are listed under Other changes. If your writable annotation tables predate source identities, adopt their rows as Adopt an annotation table written before source identities describes.

What equivalence is

A score row is stamped with a lineage ID, a hash of what produced it. Before 0.3.0, two runs reused each other's rows only when their IDs matched or a caller accepted the older ID explicitly. Moving a model to a republished container image therefore made every existing row a cache miss.

From 0.3.0, each lineage also has a LineageDescriptor: the unhashed inputs of its ID, kept in a table beside the lineage catalog. A binding declares which descriptors are scientifically the same with equivalence_projection. When scores are read, the store resolves the current lineage plus every registered lineage whose descriptor projects equal to it. Changing a binding's rules changes which rows match on the next read. It never rewrites a row.

These stay byte-for-byte the same, so nothing your application stores or compares exactly needs migrating:

  • every lineage ID that 0.2.1 computes for a configuration, except Borzoi's, Sei's, and the ones listed under Other changes;
  • PluginRunIdentity serialization, including altar_api_version, and ScoreLineage.to_json();
  • the catalog table, its two columns and its lineage_json rows;
  • plan snapshots: scoring_plan_from_dict decodes a snapshot written by 0.1.0 or 0.2.x and re-serializes it identically;
  • the return types of existing calls. prepare_unscored_table still returns an int.

New public names

Name Module Use
LineageDescriptor altar.models The unhashed inputs of a lineage ID. verify(identity) checks it against a run identity.
lineage_equivalence_key(plugin, descriptor) altar.models The hash two equivalent lineages share.
ModelPlugin.scoring_lineage_descriptor(...), .scoring_lineage_descriptor_for(request) altar.models Derive a descriptor. _for takes the same ScoringRequest as build_scoring_plan.
ModelPlugin.scoring_runtime_identities(configuration) altar.models Binding hook: the plan's runtime identities, such as {"model_runtime": image}.
ModelPlugin.equivalence_projection, .equivalence_rules, .equivalence_rules_id altar.models Binding hooks for equivalence. The default projection is the whole descriptor.
Resolution, ResolutionStep altar.results An ordered tuple of ScoreLineages with .lineage_ids, .steps (CURRENT, EQUIVALENT, ACCEPTED) and .rules_id.
ScoreReusePolicy.exactly(ids) altar.results A frozen resolution: reads exactly ids, in order.
ScoreStore.resolve_lineages(...) altar.results Resolve without reading scores.
ScoreStore.record_lineage_descriptor(identity, descriptor) altar.results Verify and record one descriptor. The lineage need not be registered.
ScoreStore.attach_lineage_descriptors(pairs) altar.results Backfill. Returns a DescriptorAttachmentReport.
ScoreStore.lineages_without_descriptors(plugin_id) altar.results Coverage check for a backfill.
ScoreStore.ensure_descriptor_table() altar.results Create the descriptor table explicitly.
DescriptorTableNotProvisioned, DescriptorTableNotProvisionedWarning altar.results Raised or warned when BigQuery's descriptor table is missing.
ScoredModel.lineage_descriptor, ScoredModel.equivalence altar.results Inputs to materialize. equivalence is the model's plugin when it is a full ModelPlugin.
ScoringPreparation.resolution altar.scoring The resolution prepare_scoring_batches used for its primary, detail and artifact checks.

Resolving calls (get_missing_scores, read_selected_scores, prepare_unscored_table, export_unscored) take optional lineage_descriptor= and equivalence= arguments. Writing calls (primary_write, promote_staged_scores, add_scores_frame, register_score_lineage) take an optional lineage_descriptor=. The score reuse guide documents each call.

Provision the descriptor table

The descriptor table is named <catalog>__descriptors: {lineage_table_id}__descriptors in BigQuery and _altar_score_lineages__descriptors in SQLite. A missing table reads as empty, so the upgrade itself causes no cache miss.

SQLite creates the table on the first descriptor it records. From 0.3.0, prepare_scoring_batches, run_scoring and ingest_scoring_results record the run's descriptor, so the first engine run against a SQLite store that keeps lineages adds this table.

BigQuery never creates it implicitly. Provision it before any 0.3.0 code runs, with deletion protection:

score_lineage_id STRING NOT NULL
plugin_id        STRING NOT NULL
descriptor_json  STRING NOT NULL
recorded_at      TIMESTAMP NOT NULL
CLUSTER BY plugin_id, score_lineage_id

Extra nullable columns are accepted. If you do not provision tables yourself, call await store.ensure_descriptor_table() once. Until the table exists:

  • every read treats it as empty, silently: resolutions find no equivalent lineages, and nothing warns;
  • an explicit record_lineage_descriptor or attach_lineage_descriptors raises DescriptorTableNotProvisioned, a CapabilityNotSupported whose table names the table;
  • only Altar's engine paths (prepare_scoring_batches, run_scoring, ingest_scoring_results) warn: they score and write as before without recording, and emit one DescriptorTableNotProvisionedWarning per store instance. An application that turns warnings into errors fails there instead.

Freeze the lineages a job reads

An application that accepts a job and reads its scores back later resolves the lineages once, when it accepts the job, and reads with exactly that resolution afterwards:

from altar.results import ScoreReusePolicy

plan = plugin.build_scoring_plan(request)
descriptor = plugin.scoring_lineage_descriptor_for(request)
await store.record_lineage_descriptor(plan.plugin_identity, descriptor)

resolution = await store.resolve_lineages(
    plan.plugin_identity,
    plugin.score_columns(),
    ScoreReusePolicy((history_lineage_id,)),  # explicit acceptances, as before
    lineage_descriptor=descriptor,
    equivalence=plugin,
)
frozen = ScoreReusePolicy.exactly(resolution.lineage_ids)
# Store resolution.lineage_ids, resolution.steps and resolution.rules_id with the job.

missing_count = await store.prepare_unscored_table(
    dest_table="`project.dataset.job_missing`",
    variants_relation="`project.dataset.job_candidates`",
    model_id=model_id,
    plugin_identity=plan.plugin_identity,
    columns=plugin.score_columns(),
    max_variants_per_batch=25_000,
    reuse_policy=frozen,
)

The resolution lists the current lineage, then registered equivalents by ID, then the policy's accepted IDs. Equivalents rank before explicit acceptances, so a variant scored under both an equivalent image and a legacy history lineage reads the equivalent row.

Pass the descriptor in memory for the first job of a new configuration: its lineage is neither registered nor recorded yet. The store verifies an in-memory descriptor against the identity and refuses a mismatch.

Read the job back with frozen, for example ScoredModel(..., reuse_policy=frozen). A frozen read needs no descriptor. Its first ID must be the run's current lineage, which need not be registered, so an all-hit job can still be assembled. Every other ID must be registered. A selection frozen before 0.3.0 maps to ScoreReusePolicy.exactly((current, *accepted)). Replay a job by comparing against its frozen resolution rather than resolving again, because a recorded descriptor or a rule change can change a fresh resolution.

Resolution is a tuple of ScoreLineages, so code that passes the result of a resolve on as lineages keeps working.

Ingestion needs no descriptor when accepting the job recorded it. A path that writes without accepting a job, such as an evaluation ingestion, records nothing; cover its lineages with the backfill.

Backfill existing lineages

Lineages written before 0.3.0 have no descriptor, so they match nothing by equivalence until one is attached. Derive (identity, descriptor) pairs from every registration you hold and attach them:

pairs = []
for request in requests_for_every_published_registration:
    plan = plugin.build_scoring_plan(request)
    pairs.append((plan.plugin_identity, plugin.scoring_lineage_descriptor_for(request)))

report = await store.attach_lineage_descriptors(pairs)
report.attached  # registered lineages that now have a descriptor
report.unregistered  # recorded, and take part once a write registers them
report.failed  # pairs whose descriptor does not reproduce the identity; nothing was recorded

remaining = await store.lineages_without_descriptors(plugin.manifest.plugin_id)

Attaching is idempotent, and a wrong configuration cannot attach anything because its recomputed lineage ID or configuration hash does not match. The backfill is complete when every lineage in remaining is attached or on a reviewed list of lineages that cannot be attached, such as one whose registration no longer exists. A missed lineage costs rescoring, not wrong results.

Imported lineages have no run identity and never appear in remaining. Keep accepting them explicitly with ScoreReusePolicy.

Some ChromBPNet lineage IDs computed by 0.1.0 differ from 0.2.1's: configurations on 93719b2e and 6fbf5d2e (listed in 0.2.0) and on 935bdc0e (listed in 0.2.1). The 0.3.0 plan builder reproduces the 0.2.1 IDs, so those older lineages stay in remaining until you attach a descriptor that reproduces them or review them as unattachable.

ChromBPNet: 9adea64e joins the default image's class

ChromBPNet's equivalence class altar-chrombpnet/class-1 now lists 9adea64e beside the default image 935bdc0e. Once a 9adea64e lineage's descriptor is attached, its rows are hits for the same configuration on any image in the class. Rows written on another image in the class become hits for 9adea64e once their lineage has a descriptor.

The validation record (runtimes/chrombpnet/validation/l40s-k562-dnase-9adea64e-equivalence.json) compared 254,917 scored rows at both batch sizes. Every row predicted in a full model batch is bit-identical to 935bdc0e, and jsd differs by at most 8.2e-5. Rows in a task's last, short model batch differ by up to 1.35e-3 logfc, which misses the pre-registered 1e-3 tolerance. The equivalence listing accepts a 1.5e-3 logfc bound for those short-batch rows, recorded as the record's accepted_exception. If you do not accept this bound, leave your 9adea64e lineages without descriptors.

An application upgrading from 0.1.0 also picks up the 0.2.0 batch-size change: the binding's -b moved from 250,000 to 25,000. 9adea64e predates final-batch padding, so short-batch rows written after the upgrade can differ by up to about 1e-3 under the same lineage ID. Upgrade after in-flight jobs drain, or accept mixed batch sizes in one lineage.

External model bindings

Implement scoring_runtime_identities(configuration) and call it from your plan builder:

def scoring_runtime_identities(self, configuration: ModelConfiguration) -> dict[str, str]:
    return {"model_runtime": self.parse_configuration_as(configuration, MyConfiguration).image}

Without it, scoring_lineage_descriptor_for raises CapabilityNotSupported for a binding that declares runtimes, so a caller cannot derive descriptors from a request. Altar's engine paths still derive a descriptor from the built plan when its identity is scoring_run_identity of the request's inputs.

To declare equivalence, override equivalence_projection(descriptor) and return the descriptor's to_dict() with the equivalent parts replaced by a shared label. Return the rule data from equivalence_rules(), so equivalence_rules_id changes when the rules do, and bump equivalence_rules_format when only the projection code changes. A projection may not drop or change plugin_id, output_selection_hash, schema_hash, genome_build or genome_hash, and may map plugin_version only as equivalence_version_bridges declares. Pin the rules ID and your fixtures' lineage_equivalence_key values in a golden test.

Variant and training assemblies

0.3.0 lets a model trained on one assembly score variants on another, such as a ChromBPNet model trained on mouse (mm10) data scoring human (hg38) variants. This section lists what changes for an existing hg38 deployment and what to do before scoring another assembly. Variant identity and normalization explains the concepts.

What stays the same

  • Every lineage ID, serialized run identity and configuration hash of an existing configuration. Declaring training_genome_build equal to the assembly a model scores on keeps them too, so you can declare it on every registration without re-keying a cache.
  • Same-assembly scoring, result assembly and prioritization on hg38, with declared or undeclared sources.

What to change

Change Who is affected What to do
Genome builds Altar writes are canonical: GRCh38, GRCh38.p14 and GRCh38_no_alt_analysis_set_... are recorded as hg38, even when the caller passed them explicitly Code that compares an ingest manifest's reference_build, a row's genome, or a run identity's genome_build with its own label Compare with same_genome_build(recorded, expected), not ==
Run inputs spelled GRCh38 record hg38 Callers whose run inputs used the GRC spelling Expect the hg38 lineage IDs, or keep using hg38
A callset union requires one recorded label across shards Anyone who reuses shard ingests across the upgrade Re-ingest older shards before a union
A source without an identity serves hg38 only, on every other assembly, known or not Tables that declare no identity, used to score another assembly Declare identity=CONTRACT.identity(release=..., genome_build=...) on each table
A source on another assembly raises only when a model needs it and a source on the variants' assembly could exist; otherwise it is dropped with a warning Code that relied on AnnotationGenomeBuildError for every mismatch Expect AnnotationUnavailableWarning for sources no model needs, and for contracts whose classes all leave the assembly out
ChromBPNet configuration schema 0.0.3 Deployments that save configurations and read them with an older Altar Upgrade every reader first (see Roll out the configuration schema); a saved 0.0.2 configuration still loads unchanged
ScoreStoreContract passes identity= to make_annotation_source A store package that overrides that fixture Accept identity and pass it to the source
contract_model_identity() and contract_model_descriptor() default to CONTRACT_MODEL_GENOME_BUILD (synthetic-v1), where a source without an identity is not joined A store package whose own tests join identity-less sources to the contract model Pass genome_build="hg38" to both, or declare an identity on the sources
Training-coordinate fields are null across assemblies Code that loads a runtime's output without ingest_scoring_results Null the columns plugin.unavailable_score_fields(configuration, run_inputs) returns

Roll out the configuration schema

ChromBPNet configurations are saved at schema 0.0.3 from 0.3.0 on. 0.3.0 loads a saved 0.0.2 configuration unchanged, but an earlier Altar rejects a 0.0.3 one. Upgrade every process that reads model configurations, such as workers, workflow runners and API servers, before any process saves a configuration with 0.3.0. Until then, a configuration re-saved by an upgraded writer cannot be loaded by a reader that has not been upgraded.

Before scoring another assembly

  1. Declare training_genome_build on every model not trained on hg38. ChromBPNet preparation reads the training assembly's reference, and only a declared training assembly lets Altar reject preparation on the wrong one.
  2. Prepare such a model with run inputs on its training assembly.
  3. Ingest the variants with --genome-build, and score them with run inputs on the variants' assembly.
  4. Declare an identity on every annotation source, so result assembly knows which assembly each one serves.

Custom score stores

A store that subclasses a core store inherits the new behavior. A store that overrides a resolving or writing method must accept the new keyword arguments: the default materialize passes lineage_descriptor= and equivalence= to read_selected_scores.

ScoreStoreContract checks descriptors, resolution and frozen policies when the store's test provides a lineage_store fixture. A store that requires its descriptor table to be provisioned sets creates_descriptor_table_on_write = False. The descriptor table's lifecycle tests run when the test also provides the descriptor_table_exists and provision_descriptor_table fixtures, and skip otherwise.

Adopt an annotation table written before source identities

Source identities arrived in 0.2.0. A writable annotation table written by 0.1.0, or loaded by your own loaders, holds rows with no identity. A scoped source neither serves nor counts such rows, so it needs a one-time backfill. Keep a writable annotation table current across releases describes scoped sources.

SQLite

Open the file with the release that built its rows as identity=, and identity_scoped=True. The source refuses to read or write the file until you call backfill_annotation_identity(), which records the identity in the file and returns IdentityBackfill(stamped=<rows adopted>, deleted=0).

BigQuery

A table without identities has no _altar_annotation_identity column, so a scoped source's queries against it fail until ensure_schema() adds the column. The existing rows then have a NULL identity. Until you run the backfill below, deduplication reports every variant as unannotated, and your next job annotates the whole table again.

These steps adopt a BigQuery table with the columns variant_id, genome, chr, pos, ref, alt plus annotation columns, clustered on genome, chr, variant_id:

  1. Find the release that built the existing rows, usually the gene annotation and cCRE releases your pipeline used. If the table mixes rows from several releases, you cannot adopt them under one identity; start a new table.
  2. Pause every job that writes to the table.
  3. Configure the writer with that old release, identity_scoped=True, and the table's stored build as genome_default. Call ensure_schema(). It adds the nullable _altar_annotation_identity STRING column, and adds empty genome, chr, pos, ref and alt columns to a table without them. It leaves existing rows and clustering as they are.
  4. Count and delete the rows whose variant_id is not canonical, such as chr9:21980945:C:C (REF equal to ALT, which some ingest paths wrote) or 9:100:C:T. No Altar read or write reaches them. Count them first:

    SELECT COUNT(*) AS non_canonical
    FROM `my-project.annotations.variant_regions`
    WHERE NOT COALESCE(
      REGEXP_CONTAINS(variant_id, '^chr[^:]+:[1-9][0-9]*:[ACGTN]+:[ACGTN]+$')
      AND SPLIT(variant_id, ':')[SAFE_OFFSET(2)] != SPLIT(variant_id, ':')[SAFE_OFFSET(3)],
      FALSE)
    

    Then delete them with the same condition:

    DELETE FROM `my-project.annotations.variant_regions`
    WHERE NOT COALESCE(
      REGEXP_CONTAINS(variant_id, '^chr[^:]+:[1-9][0-9]*:[ACGTN]+:[ACGTN]+$')
      AND SPLIT(variant_id, ':')[SAFE_OFFSET(2)] != SPLIT(variant_id, ':')[SAFE_OFFSET(3)],
      FALSE)
    

    This condition is looser than Altar's canonical check, so it never deletes a row Altar can serve. It can miss an alias such as chrMT:5:A:T; the backfills in steps 5 and 6 count any it misses. 5. If any row has a NULL genome, call backfill_identity_columns(). In one BigQuery transaction it stamps each row that has a canonical variant_id with genome_default and the locus columns, keeping one row per variant. It deletes a repeated row, and a row whose variant already has a row of that build. It returns the counts as IdentityBackfill(stamped=..., deleted=..., non_canonical=...). non_canonical counts the rows it left with a NULL genome; if it is not 0, delete them. After this call they are exactly the rows with no genome:

    DELETE FROM `my-project.annotations.variant_regions`
    WHERE genome IS NULL
    
  5. Call backfill_annotation_identity(). It writes the identity hash into every row of genome_default's build that has none. It keeps one row per variant, and deletes repeated rows and rows already superseded by a row with that hash. Among repeated rows it keeps the newest by created_at when the table has that column, then the first by the row's JSON text, so a rerun picks the same row. It returns IdentityBackfill(stamped=..., deleted=..., non_canonical=...), and a second call changes nothing. non_canonical counts the rows it left unstamped; after step 4 it is usually 0. If it is not, delete those rows, which are then exactly the rows of the build with no identity:

    DELETE FROM `my-project.annotations.variant_regions`
    WHERE genome = 'GRCh38' AND _altar_annotation_identity IS NULL
    
  6. Check the result. For the genome_default build, this query should show only your identity's hash and no NULL row:

    SELECT genome, _altar_annotation_identity, COUNT(*) AS n
    FROM `my-project.annotations.variant_regions`
    GROUP BY 1, 2
    

    Also check that every row's chr is spelled as its variant_id prefix. Rows written by your own loaders may store 1 for chr1, and get_unannotated_variants(..., chromosomes=...) and export_unannotated filter on the stored chr, so they would count such a row as missing. This query should return 0:

    SELECT COUNT(*) AS mismatched
    FROM `my-project.annotations.variant_regions`
    WHERE chr != SPLIT(variant_id, ':')[SAFE_OFFSET(0)]
    

    If it does not, rewrite those rows' locus from the variant_id:

    UPDATE `my-project.annotations.variant_regions`
    SET chr = SPLIT(variant_id, ':')[SAFE_OFFSET(0)]
    WHERE chr != SPLIT(variant_id, ':')[SAFE_OFFSET(0)]
    
  7. Give every other source object that reads the table the same old-release identity and identity_scoped=True. Add {identity_filter} to each reader's relation=.

  8. Replace any load job of your own, such as one that loads annotate JSONL from a file URI, with add_annotations_from_uri, which writes the hash and skips variants the table already has; see Load annotation files from object storage. A load path you keep must write identity.identity_hash into _altar_annotation_identity itself. A row loaded without the hash is never counted, so every job annotates that variant again and appends another row.
  9. Resume ingest.
import asyncio

from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT


async def adopt_existing_rows() -> None:
    with BigQueryAnnotationSource.connect(
        "my-project.annotations.variant_regions",
        list(REGION_ANNOTATION_CONTRACT.columns),
        name="regions",
        genome_default="GRCh38",
        identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v46-screen-v4", genome_build="hg38"),
        identity_scoped=True,
    ) as regions:
        await regions.ensure_schema()
        print(await regions.backfill_annotation_identity())


asyncio.run(adopt_existing_rows())

The backfill states that every existing row came from the configured release. Check that before you run it: adopting rows under a newer release than the one that built them serves old values under the new identity.

The backfill runs as one BigQuery transaction. It copies the unstamped rows to a temporary table, deletes them, and inserts them back with the identity. When most of the table is unstamped, it rewrites all of the table's storage. On-demand pricing bills about three times the table's logical size: a 25-million-row table of 10 GB bills about 30 GB. BigQuery re-clusters the rewritten rows in the background at no charge. The replaced storage stays in time travel and fail-safe for their configured windows, which datasets on physical storage billing pay for.

Roll back

Nothing 0.3.0 writes changes a catalog row, a score row or a plan snapshot, so you can pin back to 0.2.x or 0.1.0 and keep working. The older version ignores the descriptor table. It loses equivalence: rows written by 0.3.0 under a new image's lineage are not hits for an older version running the old image. Selections frozen in the 0.3.0 format are yours to map back.

Two things 0.3.0 writes do not read back on an older version. A ChromBPNet configuration saved by 0.3.0 is at configuration schema 0.0.3, which earlier releases refuse. Variant ingestion records hg38 where it recorded the caller's GRCh38. Variant and training assemblies describes both.

Other changes

Every public signature that changed for lineage equivalence only gained optional keyword arguments, and ScoreReusePolicy gained an exact field that defaults to False. Engine runs record descriptors, and Altar's engine paths warn once per store instance when a BigQuery descriptor table is missing; reads treat a missing table as empty without warning.

Altar's own calls into overridable store methods (read_selected_scores from the default materialize, promote_staged_scores from a BigQuery write, export_unscored from export_unscored_checked) pass the new lineage_descriptor= and equivalence= arguments only when they are set, and materialize passes equivalence= only to stores that keep lineages, so a store overriding those methods with their 0.2.x signatures keeps working until it uses equivalence.

0.3.0 also changes these, each described under Breaking in the 0.3.0 release notes:

  • Genome builds are recorded canonically, and annotation sources join only variants on their own assembly. See Variant and training assemblies.
  • Borzoi scores move: the variant now sits where the official scorer places it. Borzoi's plugin and result schema are 0.0.2, so its lineages are new and stored Borzoi scores are not reused.
  • Sei's result schema is 0.0.3, so its lineages are new.
  • AlphaGenome's track_scores detail table is at 0.0.2. Its ontology_terms and scorers are sorted, so a configuration that listed them in another order gets a new lineage.
  • Wide exports end with two more columns, prioritized_model_ids and prioritized_source_names. Shards exported before the upgrade lack them.
  • BigQueryScoreStore.prepare_unscored_table sets aside a candidate whose variant_id is not the canonical ID of its locus, and writes its misses through a scratch table. count_unscored is new, so a subclass that defines it now overrides it.
  • BigQueryScoreStore.ensure_schema runs no DDL when the table already has every column.
  • A container plan must stage its batch variant file, and a row for a variant outside the batch raises ScoringOutputError.