Upgrade from 0.2 to 0.3¶
Altar 0.3.0 adds lineage equivalence: when a store reads scores, it also reuses the rows of older lineages that the binding declares scientifically the same. Most of the steps below are for an application that keeps scores in an Altar score store from its own code: what to provision, which calls to change, and how to backfill existing lineages.
0.3.0 also lets a model score variants on another assembly than the one it was trained on, which changes how Altar records genome builds. Variant and training assemblies covers that upgrade. The rest of 0.3.0's breaking changes are listed under Other changes. If your writable annotation tables predate source identities, adopt their rows as Adopt an annotation table written before source identities describes.
What equivalence is¶
A score row is stamped with a lineage ID, a hash of what produced it. Before 0.3.0, two runs reused each other's rows only when their IDs matched or a caller accepted the older ID explicitly. Moving a model to a republished container image therefore made every existing row a cache miss.
From 0.3.0, each lineage also has a LineageDescriptor: the unhashed inputs of its ID, kept in a table beside the
lineage catalog. A binding declares which descriptors are scientifically the same with equivalence_projection.
When scores are read, the store resolves the current lineage plus every registered lineage whose descriptor
projects equal to it. Changing a binding's rules changes which rows match on the next read. It never rewrites a
row.
These stay byte-for-byte the same, so nothing your application stores or compares exactly needs migrating:
- every lineage ID that 0.2.1 computes for a configuration, except Borzoi's, Sei's, and the ones listed under Other changes;
PluginRunIdentityserialization, includingaltar_api_version, andScoreLineage.to_json();- the catalog table, its two columns and its
lineage_jsonrows; - plan snapshots:
scoring_plan_from_dictdecodes a snapshot written by 0.1.0 or 0.2.x and re-serializes it identically; - the return types of existing calls.
prepare_unscored_tablestill returns anint.
New public names¶
| Name | Module | Use |
|---|---|---|
LineageDescriptor |
altar.models |
The unhashed inputs of a lineage ID. verify(identity) checks it against a run identity. |
lineage_equivalence_key(plugin, descriptor) |
altar.models |
The hash two equivalent lineages share. |
ModelPlugin.scoring_lineage_descriptor(...), .scoring_lineage_descriptor_for(request) |
altar.models |
Derive a descriptor. _for takes the same ScoringRequest as build_scoring_plan. |
ModelPlugin.scoring_runtime_identities(configuration) |
altar.models |
Binding hook: the plan's runtime identities, such as {"model_runtime": image}. |
ModelPlugin.equivalence_projection, .equivalence_rules, .equivalence_rules_id |
altar.models |
Binding hooks for equivalence. The default projection is the whole descriptor. |
Resolution, ResolutionStep |
altar.results |
An ordered tuple of ScoreLineages with .lineage_ids, .steps (CURRENT, EQUIVALENT, ACCEPTED) and .rules_id. |
ScoreReusePolicy.exactly(ids) |
altar.results |
A frozen resolution: reads exactly ids, in order. |
ScoreStore.resolve_lineages(...) |
altar.results |
Resolve without reading scores. |
ScoreStore.record_lineage_descriptor(identity, descriptor) |
altar.results |
Verify and record one descriptor. The lineage need not be registered. |
ScoreStore.attach_lineage_descriptors(pairs) |
altar.results |
Backfill. Returns a DescriptorAttachmentReport. |
ScoreStore.lineages_without_descriptors(plugin_id) |
altar.results |
Coverage check for a backfill. |
ScoreStore.ensure_descriptor_table() |
altar.results |
Create the descriptor table explicitly. |
DescriptorTableNotProvisioned, DescriptorTableNotProvisionedWarning |
altar.results |
Raised or warned when BigQuery's descriptor table is missing. |
ScoredModel.lineage_descriptor, ScoredModel.equivalence |
altar.results |
Inputs to materialize. equivalence is the model's plugin when it is a full ModelPlugin. |
ScoringPreparation.resolution |
altar.scoring |
The resolution prepare_scoring_batches used for its primary, detail and artifact checks. |
Resolving calls (get_missing_scores, read_selected_scores, prepare_unscored_table, export_unscored) take
optional lineage_descriptor= and equivalence= arguments. Writing calls (primary_write,
promote_staged_scores, add_scores_frame, register_score_lineage) take an optional lineage_descriptor=. The
score reuse guide documents each call.
Provision the descriptor table¶
The descriptor table is named <catalog>__descriptors: {lineage_table_id}__descriptors in BigQuery and
_altar_score_lineages__descriptors in SQLite. A missing table reads as empty, so the upgrade itself causes no
cache miss.
SQLite creates the table on the first descriptor it records. From 0.3.0, prepare_scoring_batches,
run_scoring and ingest_scoring_results record the run's descriptor, so the first engine run against a
SQLite store that keeps lineages adds this table.
BigQuery never creates it implicitly. Provision it before any 0.3.0 code runs, with deletion protection:
score_lineage_id STRING NOT NULL
plugin_id STRING NOT NULL
descriptor_json STRING NOT NULL
recorded_at TIMESTAMP NOT NULL
CLUSTER BY plugin_id, score_lineage_id
Extra nullable columns are accepted. If you do not provision tables yourself, call await
store.ensure_descriptor_table() once. Until the table exists:
- every read treats it as empty, silently: resolutions find no equivalent lineages, and nothing warns;
- an explicit
record_lineage_descriptororattach_lineage_descriptorsraisesDescriptorTableNotProvisioned, aCapabilityNotSupportedwhosetablenames the table; - only Altar's engine paths (
prepare_scoring_batches,run_scoring,ingest_scoring_results) warn: they score and write as before without recording, and emit oneDescriptorTableNotProvisionedWarningper store instance. An application that turns warnings into errors fails there instead.
Freeze the lineages a job reads¶
An application that accepts a job and reads its scores back later resolves the lineages once, when it accepts the job, and reads with exactly that resolution afterwards:
from altar.results import ScoreReusePolicy
plan = plugin.build_scoring_plan(request)
descriptor = plugin.scoring_lineage_descriptor_for(request)
await store.record_lineage_descriptor(plan.plugin_identity, descriptor)
resolution = await store.resolve_lineages(
plan.plugin_identity,
plugin.score_columns(),
ScoreReusePolicy((history_lineage_id,)), # explicit acceptances, as before
lineage_descriptor=descriptor,
equivalence=plugin,
)
frozen = ScoreReusePolicy.exactly(resolution.lineage_ids)
# Store resolution.lineage_ids, resolution.steps and resolution.rules_id with the job.
missing_count = await store.prepare_unscored_table(
dest_table="`project.dataset.job_missing`",
variants_relation="`project.dataset.job_candidates`",
model_id=model_id,
plugin_identity=plan.plugin_identity,
columns=plugin.score_columns(),
max_variants_per_batch=25_000,
reuse_policy=frozen,
)
The resolution lists the current lineage, then registered equivalents by ID, then the policy's accepted IDs. Equivalents rank before explicit acceptances, so a variant scored under both an equivalent image and a legacy history lineage reads the equivalent row.
Pass the descriptor in memory for the first job of a new configuration: its lineage is neither registered nor recorded yet. The store verifies an in-memory descriptor against the identity and refuses a mismatch.
Read the job back with frozen, for example ScoredModel(..., reuse_policy=frozen). A frozen read needs no
descriptor. Its first ID must be the run's current lineage, which need not be registered, so an all-hit job can
still be assembled. Every other ID must be registered. A selection frozen before 0.3.0 maps to
ScoreReusePolicy.exactly((current, *accepted)). Replay a job by comparing against its frozen resolution rather
than resolving again, because a recorded descriptor or a rule change can change a fresh resolution.
Resolution is a tuple of ScoreLineages, so code that passes the result of a resolve on as lineages keeps
working.
Ingestion needs no descriptor when accepting the job recorded it. A path that writes without accepting a job, such as an evaluation ingestion, records nothing; cover its lineages with the backfill.
Backfill existing lineages¶
Lineages written before 0.3.0 have no descriptor, so they match nothing by equivalence until one is attached.
Derive (identity, descriptor) pairs from every registration you hold and attach them:
pairs = []
for request in requests_for_every_published_registration:
plan = plugin.build_scoring_plan(request)
pairs.append((plan.plugin_identity, plugin.scoring_lineage_descriptor_for(request)))
report = await store.attach_lineage_descriptors(pairs)
report.attached # registered lineages that now have a descriptor
report.unregistered # recorded, and take part once a write registers them
report.failed # pairs whose descriptor does not reproduce the identity; nothing was recorded
remaining = await store.lineages_without_descriptors(plugin.manifest.plugin_id)
Attaching is idempotent, and a wrong configuration cannot attach anything because its recomputed lineage ID or
configuration hash does not match. The backfill is complete when every lineage in remaining is attached or on a
reviewed list of lineages that cannot be attached, such as one whose registration no longer exists. A missed
lineage costs rescoring, not wrong results.
Imported lineages have no run identity and never appear in remaining. Keep accepting them explicitly with
ScoreReusePolicy.
Some ChromBPNet lineage IDs computed by 0.1.0 differ from 0.2.1's: configurations on 93719b2e and 6fbf5d2e
(listed in 0.2.0) and on 935bdc0e (listed in 0.2.1). The 0.3.0 plan builder reproduces the 0.2.1 IDs, so those
older lineages stay in remaining until you attach a descriptor that reproduces them or review them as
unattachable.
ChromBPNet: 9adea64e joins the default image's class¶
ChromBPNet's equivalence class altar-chrombpnet/class-1 now lists 9adea64e beside the default image
935bdc0e. Once a 9adea64e lineage's descriptor is attached, its rows are hits for the same configuration on
any image in the class. Rows written on another image in the class become hits for 9adea64e once their lineage
has a descriptor.
The validation record (runtimes/chrombpnet/validation/l40s-k562-dnase-9adea64e-equivalence.json) compared
254,917 scored rows at both batch sizes. Every row predicted in a full model batch is bit-identical to
935bdc0e, and jsd differs by at most 8.2e-5. Rows in a task's last, short model batch differ by up to 1.35e-3
logfc, which misses the pre-registered 1e-3 tolerance. The equivalence listing accepts a 1.5e-3 logfc bound
for those short-batch rows, recorded as the record's accepted_exception. If you do not accept this bound, leave
your 9adea64e lineages without descriptors.
An application upgrading from 0.1.0 also picks up the 0.2.0 batch-size change: the binding's -b moved from 250,000 to
25,000. 9adea64e predates final-batch padding, so short-batch rows written after the upgrade can differ by up to
about 1e-3 under the same lineage ID. Upgrade after in-flight jobs drain, or accept mixed batch sizes in one
lineage.
External model bindings¶
Implement scoring_runtime_identities(configuration) and call it from your plan builder:
def scoring_runtime_identities(self, configuration: ModelConfiguration) -> dict[str, str]:
return {"model_runtime": self.parse_configuration_as(configuration, MyConfiguration).image}
Without it, scoring_lineage_descriptor_for raises CapabilityNotSupported for a binding that declares
runtimes, so a caller cannot derive descriptors from a request. Altar's engine paths still derive a descriptor from
the built plan when its identity is scoring_run_identity of the request's inputs.
To declare equivalence, override equivalence_projection(descriptor) and return the descriptor's to_dict()
with the equivalent parts replaced by a shared label. Return the rule data from equivalence_rules(), so
equivalence_rules_id changes when the rules do, and bump equivalence_rules_format when only the projection code
changes. A projection may not drop or change plugin_id, output_selection_hash, schema_hash, genome_build
or genome_hash, and may map plugin_version only as equivalence_version_bridges declares. Pin the rules ID
and your fixtures' lineage_equivalence_key values in a golden test.
Variant and training assemblies¶
0.3.0 lets a model trained on one assembly score variants on another, such as a ChromBPNet model trained on mouse
(mm10) data scoring human (hg38) variants. This section lists what changes for an existing hg38 deployment and
what to do before scoring another assembly. Variant identity and normalization explains
the concepts.
What stays the same¶
- Every lineage ID, serialized run identity and configuration hash of an existing configuration. Declaring
training_genome_buildequal to the assembly a model scores on keeps them too, so you can declare it on every registration without re-keying a cache. - Same-assembly scoring, result assembly and prioritization on hg38, with declared or undeclared sources.
What to change¶
| Change | Who is affected | What to do |
|---|---|---|
Genome builds Altar writes are canonical: GRCh38, GRCh38.p14 and GRCh38_no_alt_analysis_set_... are recorded as hg38, even when the caller passed them explicitly |
Code that compares an ingest manifest's reference_build, a row's genome, or a run identity's genome_build with its own label |
Compare with same_genome_build(recorded, expected), not == |
Run inputs spelled GRCh38 record hg38 |
Callers whose run inputs used the GRC spelling | Expect the hg38 lineage IDs, or keep using hg38 |
| A callset union requires one recorded label across shards | Anyone who reuses shard ingests across the upgrade | Re-ingest older shards before a union |
A source without an identity serves hg38 only, on every other assembly, known or not |
Tables that declare no identity, used to score another assembly | Declare identity=CONTRACT.identity(release=..., genome_build=...) on each table |
| A source on another assembly raises only when a model needs it and a source on the variants' assembly could exist; otherwise it is dropped with a warning | Code that relied on AnnotationGenomeBuildError for every mismatch |
Expect AnnotationUnavailableWarning for sources no model needs, and for contracts whose classes all leave the assembly out |
| ChromBPNet configuration schema 0.0.3 | Deployments that save configurations and read them with an older Altar | Upgrade every reader first (see Roll out the configuration schema); a saved 0.0.2 configuration still loads unchanged |
ScoreStoreContract passes identity= to make_annotation_source |
A store package that overrides that fixture | Accept identity and pass it to the source |
contract_model_identity() and contract_model_descriptor() default to CONTRACT_MODEL_GENOME_BUILD (synthetic-v1), where a source without an identity is not joined |
A store package whose own tests join identity-less sources to the contract model | Pass genome_build="hg38" to both, or declare an identity on the sources |
| Training-coordinate fields are null across assemblies | Code that loads a runtime's output without ingest_scoring_results |
Null the columns plugin.unavailable_score_fields(configuration, run_inputs) returns |
Roll out the configuration schema¶
ChromBPNet configurations are saved at schema 0.0.3 from 0.3.0 on. 0.3.0 loads a saved 0.0.2 configuration unchanged, but an earlier Altar rejects a 0.0.3 one. Upgrade every process that reads model configurations, such as workers, workflow runners and API servers, before any process saves a configuration with 0.3.0. Until then, a configuration re-saved by an upgraded writer cannot be loaded by a reader that has not been upgraded.
Before scoring another assembly¶
- Declare
training_genome_buildon every model not trained on hg38. ChromBPNet preparation reads the training assembly's reference, and only a declared training assembly lets Altar reject preparation on the wrong one. - Prepare such a model with run inputs on its training assembly.
- Ingest the variants with
--genome-build, and score them with run inputs on the variants' assembly. - Declare an identity on every annotation source, so result assembly knows which assembly each one serves.
Custom score stores¶
A store that subclasses a core store inherits the new behavior. A store that overrides a resolving or writing
method must accept the new keyword arguments: the default materialize passes lineage_descriptor= and
equivalence= to read_selected_scores.
ScoreStoreContract checks descriptors, resolution and frozen policies when the store's test provides a
lineage_store fixture. A store that requires its descriptor table to be provisioned sets
creates_descriptor_table_on_write = False. The descriptor table's lifecycle tests run when the test also provides the
descriptor_table_exists and provision_descriptor_table fixtures, and skip otherwise.
Adopt an annotation table written before source identities¶
Source identities arrived in 0.2.0. A writable annotation table written by 0.1.0, or loaded by your own loaders, holds rows with no identity. A scoped source neither serves nor counts such rows, so it needs a one-time backfill. Keep a writable annotation table current across releases describes scoped sources.
SQLite¶
Open the file with the release that built its rows as identity=, and identity_scoped=True. The source refuses to read
or write the file until you call backfill_annotation_identity(), which records the identity in the file and returns
IdentityBackfill(stamped=<rows adopted>, deleted=0).
BigQuery¶
A table without identities has no _altar_annotation_identity column, so a scoped source's queries against it fail
until ensure_schema() adds the column. The existing rows then have a NULL identity. Until you run the backfill below,
deduplication reports every variant as unannotated, and your next job annotates the whole table again.
These steps adopt a BigQuery table with the columns variant_id, genome, chr, pos, ref, alt plus annotation columns,
clustered on genome, chr, variant_id:
- Find the release that built the existing rows, usually the gene annotation and cCRE releases your pipeline used. If the table mixes rows from several releases, you cannot adopt them under one identity; start a new table.
- Pause every job that writes to the table.
- Configure the writer with that old release,
identity_scoped=True, and the table's stored build asgenome_default. Callensure_schema(). It adds the nullable_altar_annotation_identity STRINGcolumn, and adds emptygenome,chr,pos,refandaltcolumns to a table without them. It leaves existing rows and clustering as they are. -
Count and delete the rows whose
variant_idis not canonical, such aschr9:21980945:C:C(REF equal to ALT, which some ingest paths wrote) or9:100:C:T. No Altar read or write reaches them. Count them first:SELECT COUNT(*) AS non_canonical FROM `my-project.annotations.variant_regions` WHERE NOT COALESCE( REGEXP_CONTAINS(variant_id, '^chr[^:]+:[1-9][0-9]*:[ACGTN]+:[ACGTN]+$') AND SPLIT(variant_id, ':')[SAFE_OFFSET(2)] != SPLIT(variant_id, ':')[SAFE_OFFSET(3)], FALSE)Then delete them with the same condition:
DELETE FROM `my-project.annotations.variant_regions` WHERE NOT COALESCE( REGEXP_CONTAINS(variant_id, '^chr[^:]+:[1-9][0-9]*:[ACGTN]+:[ACGTN]+$') AND SPLIT(variant_id, ':')[SAFE_OFFSET(2)] != SPLIT(variant_id, ':')[SAFE_OFFSET(3)], FALSE)This condition is looser than Altar's canonical check, so it never deletes a row Altar can serve. It can miss an alias such as
chrMT:5:A:T; the backfills in steps 5 and 6 count any it misses. 5. If any row has a NULLgenome, callbackfill_identity_columns(). In one BigQuery transaction it stamps each row that has a canonicalvariant_idwithgenome_defaultand the locus columns, keeping one row per variant. It deletes a repeated row, and a row whose variant already has a row of that build. It returns the counts asIdentityBackfill(stamped=..., deleted=..., non_canonical=...).non_canonicalcounts the rows it left with a NULLgenome; if it is not 0, delete them. After this call they are exactly the rows with nogenome: -
Call
backfill_annotation_identity(). It writes the identity hash into every row ofgenome_default's build that has none. It keeps one row per variant, and deletes repeated rows and rows already superseded by a row with that hash. Among repeated rows it keeps the newest bycreated_atwhen the table has that column, then the first by the row's JSON text, so a rerun picks the same row. It returnsIdentityBackfill(stamped=..., deleted=..., non_canonical=...), and a second call changes nothing.non_canonicalcounts the rows it left unstamped; after step 4 it is usually 0. If it is not, delete those rows, which are then exactly the rows of the build with no identity: -
Check the result. For the
genome_defaultbuild, this query should show only your identity's hash and no NULL row:SELECT genome, _altar_annotation_identity, COUNT(*) AS n FROM `my-project.annotations.variant_regions` GROUP BY 1, 2Also check that every row's
chris spelled as itsvariant_idprefix. Rows written by your own loaders may store1forchr1, andget_unannotated_variants(..., chromosomes=...)andexport_unannotatedfilter on the storedchr, so they would count such a row as missing. This query should return 0:SELECT COUNT(*) AS mismatched FROM `my-project.annotations.variant_regions` WHERE chr != SPLIT(variant_id, ':')[SAFE_OFFSET(0)]If it does not, rewrite those rows' locus from the
variant_id: -
Give every other source object that reads the table the same old-release identity and
identity_scoped=True. Add{identity_filter}to each reader'srelation=. - Replace any load job of your own, such as one that loads annotate JSONL from a file URI, with
add_annotations_from_uri, which writes the hash and skips variants the table already has; see Load annotation files from object storage. A load path you keep must writeidentity.identity_hashinto_altar_annotation_identityitself. A row loaded without the hash is never counted, so every job annotates that variant again and appends another row. - Resume ingest.
import asyncio
from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT
async def adopt_existing_rows() -> None:
with BigQueryAnnotationSource.connect(
"my-project.annotations.variant_regions",
list(REGION_ANNOTATION_CONTRACT.columns),
name="regions",
genome_default="GRCh38",
identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v46-screen-v4", genome_build="hg38"),
identity_scoped=True,
) as regions:
await regions.ensure_schema()
print(await regions.backfill_annotation_identity())
asyncio.run(adopt_existing_rows())
The backfill states that every existing row came from the configured release. Check that before you run it: adopting rows under a newer release than the one that built them serves old values under the new identity.
The backfill runs as one BigQuery transaction. It copies the unstamped rows to a temporary table, deletes them, and inserts them back with the identity. When most of the table is unstamped, it rewrites all of the table's storage. On-demand pricing bills about three times the table's logical size: a 25-million-row table of 10 GB bills about 30 GB. BigQuery re-clusters the rewritten rows in the background at no charge. The replaced storage stays in time travel and fail-safe for their configured windows, which datasets on physical storage billing pay for.
Roll back¶
Nothing 0.3.0 writes changes a catalog row, a score row or a plan snapshot, so you can pin back to 0.2.x or 0.1.0 and keep working. The older version ignores the descriptor table. It loses equivalence: rows written by 0.3.0 under a new image's lineage are not hits for an older version running the old image. Selections frozen in the 0.3.0 format are yours to map back.
Two things 0.3.0 writes do not read back on an older version. A ChromBPNet configuration saved by 0.3.0 is at
configuration schema 0.0.3, which earlier releases refuse. Variant ingestion records hg38 where it recorded the
caller's GRCh38. Variant and training assemblies describes both.
Other changes¶
Every public signature that changed for lineage equivalence only gained optional keyword arguments, and
ScoreReusePolicy gained an exact field that defaults to False. Engine runs record descriptors, and Altar's
engine paths warn once per store instance when a BigQuery descriptor table is missing; reads treat a missing table
as empty without warning.
Altar's own calls into overridable store methods (read_selected_scores from the default materialize,
promote_staged_scores from a BigQuery write, export_unscored from export_unscored_checked) pass the new
lineage_descriptor= and equivalence= arguments only when they are set, and materialize passes equivalence=
only to stores that keep lineages, so a store overriding those methods with their 0.2.x signatures keeps working
until it uses equivalence.
0.3.0 also changes these, each described under Breaking in the 0.3.0 release notes:
- Genome builds are recorded canonically, and annotation sources join only variants on their own assembly. See Variant and training assemblies.
- Borzoi scores move: the variant now sits where the official scorer places it. Borzoi's plugin and result schema are 0.0.2, so its lineages are new and stored Borzoi scores are not reused.
- Sei's result schema is 0.0.3, so its lineages are new.
- AlphaGenome's
track_scoresdetail table is at 0.0.2. Itsontology_termsandscorersare sorted, so a configuration that listed them in another order gets a new lineage. - Wide exports end with two more columns,
prioritized_model_idsandprioritized_source_names. Shards exported before the upgrade lack them. BigQueryScoreStore.prepare_unscored_tablesets aside a candidate whosevariant_idis not the canonical ID of its locus, and writes its misses through a scratch table.count_unscoredis new, so a subclass that defines it now overrides it.BigQueryScoreStore.ensure_schemaruns no DDL when the table already has every column.- A container plan must stage its batch variant file, and a row for a variant outside the batch raises
ScoringOutputError.