Skip to content

Prepare model resources

Some models derive files from their weights and reference data before they can score. ChromBPNet, for example, calibrates each fold against the model's peaks and builds an interval index. altar prepare (Provisional) runs a model's preparation step and puts its files in the cache, where altar run reads them. The examples prepare the K562 DNase ChromBPNet model and then score with it.

altar models <type> says whether a model needs preparation. ChromBPNet's entry ends with:

Preparation:
  Run `altar prepare spec.toml --model <section-name>` before scoring.

A model without that line needs no preparation.

Before you start

  • Docker with the NVIDIA Container Toolkit on an amd64 machine, as for scoring; see Run tasks on a GPU or a cluster. ChromBPNet's preparation needs a 24 GB-class GPU. Its image is 6.1 GB compressed.
  • The model's resources. Install the binding and fetch the official K562 DNase ensemble, which writes chrombpnet.resources.json beside the weights and peaks:

    pip install altar-chrombpnet
    altar-chrombpnet-resources fetch-official --output-dir chrombpnet-resources
    
  • The reference FASTA of the model's training build: hg38 for this model. Preparation reads the model's peaks, which are on that build.

ChromBPNet gives the model's memory and run-time measurements.

Run altar prepare

Preparation needs a spec with [run], [variants] reference_fasta, the model section and a [backend]. It needs no cohort and no store. prepare.toml:

[run]
job_id = "prepare-k562"
genome_build = "hg38"

[variants]
reference_fasta = "hg38.fa"

[models.k562-dnase]
type = "CHROMBPNET"
resource_manifest = "chrombpnet-resources/chrombpnet.resources.json"

[backend]
type = "local"

Preparation writes only to the cache: .altar in the directory you run it from, unless [run] cache_dir or the ALTAR_CACHE_DIR environment variable names another. It creates no directory for job_id.

The scoring spec can serve too. Preparation ignores its sources, stores, export and interpretation sections, and never opens [variants] path, so you can prepare before the cohort exists.

Check the plan first. --validate, Provisional like altar prepare itself, reads no reference and writes no file:

$ altar prepare prepare.toml --model k562-dnase --validate
the spec is valid; nothing was read or written. Preparation would:
  run [models.k562-dnase] (org.kundajelab.altar.model.chrombpnet) in stages: prepare
  publish 7 files of [models.k562-dnase] under .altar/staging/models/k562-dnase
{
  "k562-dnase": {
    "plugin_id": "org.kundajelab.altar.model.chrombpnet",
    "stages": [
      "prepare"
    ],
    "output_uris": [
      ".altar/staging/models/k562-dnase/fold_0_peak_distribution.npy",
      ".altar/staging/models/k562-dnase/fold_1_peak_distribution.npy",
      ".altar/staging/models/k562-dnase/fold_2_peak_distribution.npy",
      ".altar/staging/models/k562-dnase/fold_3_peak_distribution.npy",
      ".altar/staging/models/k562-dnase/fold_4_peak_distribution.npy",
      ".altar/staging/models/k562-dnase/peaks.intervals.jsonl",
      ".altar/staging/models/k562-dnase/preprocessing.manifest.json"
    ]
  }
}

Then run it:

altar prepare prepare.toml --model k562-dnase

--model names a model key, the <name> of a [models.<name>] section; repeat it to prepare several. Without it, every model in the spec must offer preparation. The command verifies the reference, runs the model's preparation tasks on the backend, and prints progress on standard error. It prints the completed stages and the published files as JSON on standard output.

Every altar prepare runs the preparation again, even when its files already exist. A file's name says nothing about whether it matches the current weights, peaks, reference or settings.

Score with the prepared files

Score with a spec that uses the same cache, the same model key, and the same resources and settings. Run it from the same directory, or set the same cache_dir; its job_id may differ. It adds the cohort and the region source that ChromBPNet's prioritization reads:

[run]
job_id = "score-k562"
genome_build = "hg38"

[variants]
path = "cohort.vcf"
reference_fasta = "hg38.fa"

[models.k562-dnase]
type = "CHROMBPNET"
resource_manifest = "chrombpnet-resources/chrombpnet.resources.json"

[backend]
type = "local"

[sources.regions]
type = "regions"
options = { data_dir = "regions-hg38" }

Genomic regions describes the region data the regions source reads.

Before it reads the cohort, the run checks the prepared files:

  • The files must be under <cache_dir>/staging. A missing one stops the run and names it.
  • altar prepare recorded the settings it ran with in <cache_dir>/preparation-state/<name>.json. The model's binding compares them with the scoring configuration. ChromBPNet compares its weights, peaks and average_reverse, and a difference stops the run. ChromBPNet's fold tasks then check the files again, including the reference they were prepared from.

altar run --validate makes the same checks. Run altar prepare again after you change a model's scientific configuration. The checks are not part of a score's identity, and stored scores stay reusable. ChromBPNet describes the setting that turns them off.

Check the result

The published files are under <cache_dir>/staging/models/<name>/, the paths --validate listed, and the record is <cache_dir>/preparation-state/<name>.json. Then check the scoring spec:

altar run --validate score.toml

It passes the prepared-file checks when the files are in place and match the configuration.

If it fails

A scoring run before preparation stops and lists the files it looked for:

altar run: error: [models.k562-dnase] reads files a preparation step writes, and .altar/staging does not have them: models/k562-dnase/fold_0_peak_distribution.npy, models/k562-dnase/fold_1_peak_distribution.npy, models/k562-dnase/fold_2_peak_distribution.npy, models/k562-dnase/fold_3_peak_distribution.npy, models/k562-dnase/fold_4_peak_distribution.npy, models/k562-dnase/peaks.intervals.jsonl. Run `altar prepare SPEC --model k562-dnase` with this spec first; https://kundajelab.github.io/altar/integrations/chrombpnet/ describes it

Recover from a failed preparation

Preparation collects each attempt's files into its own directory under <cache_dir>/preparation-attempts. Once every file is there, it marks the model's publication incomplete, copies the files into <cache_dir>/staging, and clears the mark after the last copy.

  • A preparation that fails before it publishes leaves the previous files in staging as they were.
  • A publication that fails or is interrupted leaves the mark, and every scoring run refuses the model, even one whose scores are all cached:

    altar run: error: [models.k562-dnase] preparation publication is incomplete; run `altar prepare SPEC --model k562-dnase` successfully before scoring, even when scores are already cached
    

    Run altar prepare again until it succeeds. Do not delete the mark to get past the check.

  • A failed preparation prints an altar prepare: error: line. Run it again with -v to see where it stopped.

Attempt directories are kept for diagnosis. After a successful preparation you may delete old ones; keep staging and preparation-state. A scoring run's own directory, named after its job_id, holds none of these, so deleting it keeps the prepared files.

Two rules prevent other failures:

  • Do not run preparation and scoring, or two preparations, at the same time with one cache. Use separate caches for independent work.
  • The local backend's data_root must be a directory apart from <cache_dir>/staging. A data_root inside staging, one that contains it, or a link to either is refused. The default, <cache_dir>/backend-data, is apart from it.

Fix a run that stops or warns covers the other messages.

Next