Add a conservation track¶
A conservation track is one base-wise score file, such as a UCSC bigWig, on one genome build. altar-conservation
turns each track into an annotation source that adds one column to every variant: the score at the variant's base.
phyloP 100-way on hg38 is the first track. This page adds another, using UCSC's hg38 phastCons 100-way as the
example.
A track needs:
- a contract and a
ConservationTrackentry intracks.py; - the entry in
TRACKS; - a four-line source subclass;
- an entry point;
- tests;
- a build with
altar-conservation-build --track; - a published copy, following the release runbook;
- a Hugging Face dataset card.
The position score layout that a track is read from serves hg19 and hg38, and the builder keeps chr1–22, X, Y and
M. A track on another assembly or with other contigs needs changes in altar first.
1. Declare the track¶
In bindings/conservation/altar_conservation/tracks.py, declare the track's contract, its one float column, and
its ConservationTrack:
PHASTCONS100WAY_ANNOTATIONS = AnnotationContract(
source_id="org.kundajelab.altar.annotation.phastcons100way",
schema_version="1.0.0",
columns=(AnnotationColumn(name="phastcons100way", dtype="float", label="phastCons 100-way"),),
)
"""The phastCons 100-way annotation contract: one float column, ``phastcons100way``.
The value is UCSC's phastCons score over the 100-vertebrate alignment at the variant's base: the posterior
probability, from 0 to 1, that the base is in a conserved element.
"""
PHASTCONS100WAY = ConservationTrack(
contract=PHASTCONS100WAY_ANNOTATIONS,
genome_build="hg38",
release="2015-05",
upstream_url="https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw",
upstream_md5="8df8387aba88f4ca9eacdebb2e729745",
published=_published(
PHASTCONS100WAY_ANNOTATIONS, release="2015-05", genome_build="hg38", path="phastcons100way-hg38"
),
)
"""UCSC hg38 phastCons 100-way (``hg38.phastCons100way.bw``, May 2015), published as ``phastcons100way-hg38``."""
- The column's name is the track's name. It is also the source's registry name and the builder's
--trackvalue. releaseis the upstream file's date,YYYY-MM, from UCSC's directory listing.upstream_md5is the checksum the provider publishes, here from UCSC'smd5sum.txt. The builder refuses a file that does not match.pathnames the published copy,gs://kundajelab-altar/conservation/<path>andhf://datasets/altar-genomics/<path>. Follow the naming rules: the track's name and its genome build.
2. Add it to TRACKS¶
TRACKS lists every track by name. The builder's --track choices come from it:
TRACKS: dict[str, ConservationTrack] = {track.name: track for track in (PHYLOP100WAY, PHASTCONS100WAY)}
Add the two new names to the module's __all__.
3. Subclass the source¶
In source.py, a track's source is a subclass that names the track and the registry name:
class PhastCons100wayAnnotationSource(ConservationAnnotationSource):
"""UCSC hg38 phastCons 100-way (`PHASTCONS100WAY`), registered as ``phastcons100way``."""
name = "phastcons100way"
track = PHASTCONS100WAY
The subclass inherits from_options, the published copy, altar sources' contract and the option errors.
ConservationAnnotationSource.__init_subclass__ sets supported_genome_builds to the track's genome build, so
altar run refuses the source for a cohort on another build, and results assembled from Python join it only for
variants on that build.
Export the subclass, the track and its contract from altar_conservation/__init__.py.
4. Register an entry point¶
In bindings/conservation/pyproject.toml, the entry point's key must equal the source's name:
[project.entry-points."altar.annotation_sources"]
phylop100way = "altar_conservation:PhyloP100wayAnnotationSource"
phastcons100way = "altar_conservation:PhastCons100wayAnnotationSource"
Reinstall the binding so the entry point is registered, for example with
uvx uv@0.12.17 sync --all-packages --extra test --extra kubernetes.
5. Test it¶
In bindings/conservation/tests/test_conservation.py, mirror the phyloP tests for the new track:
- the track and its published dataset:
TRACKS, the registry, the contract, both location URIs, the identity andcluster_by; supported_genome_buildsis the track's build;- a conformance class over
AnnotationSourceContract, and one overPublishedDatasetContractwithreads_bigquery_tablesreturningFalse, both reading a dataset written withwrite_contigand the track'strack_metadata; altar sourceslists the track, andaltar sources <name>shows its release, build, URI and column.
Then run the binding's tests and the gate:
6. Build the dataset¶
Download the bigWig and build from it on a server with enough memory. phyloP 100-way took 72 s and about 14 GB on a 64-core server:
curl -O https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw
altar-conservation-build --track phastcons100way --input hg38.phastCons100way.bw \
--output /scratch/phastcons100way-hg38
The builder checks the MD5, converts each primary contig in its own process, and records the track, release and genome build in every footer.
7. Publish it¶
Upload the build to its prefix and verify it, as Publish a dataset release describes, with
the phyloP notes for a local build: gcloud storage rsync --recursive to
gs://kundajelab-altar/conservation/phastcons100way-hg38, then altar pull phastcons100way --to <scratch> to check
every file. Add the track to the runbook's table of published datasets.
8. Write the dataset card¶
Dataset cards live in bindings/conservation/hf-card/; README.md there is the phyloP card. Copy it to a card
named for the new published copy, such as hf-card/phastcons100way-hg38.md, and update every track-specific value:
the front matter, the title, what the score means, the upstream file and checksum, the row count, file count and
size, the rows per contig, the footer values, the URIs, the DuckDB example, and the citation UCSC asks for. Copy the
card into the build directory as README.md, then mirror the build to Hugging Face, as step 5 of the runbook shows.
Finally, document the track for users:
- its own page under
altar/docs/integrations/, on the integration page template, like phyloP 100-way, with an entry in thenavofaltar/mkdocs.yml. Runscripts/gen_integration_pages.py --updateto fill its generated blocks and add its row to the catalog; - the binding's README.