[conda] Should conda specs be immutable references? (OR: how to make fully reproducible environments)

[jtratner via conda - Public]

Hi all,

Recently, with the conda-forge and bioconda changes, I've noticed many
conda packages changing HTTP paths, which is throwing a wrench in my
attempt to set up a "lockfile" for my conda environment. I'd like to have
no package resolution at install time (i.e., have guarantees that our
builds are identical) and be able to reproduce older environments at will.

Is there a blessed way to reference packages for posterity? (My
understanding is an environment file still has to go through package
resolution again, so it's not necessary fully reproducible?)

Right now, we are attempting to do this in the following way:

1. keep a file "conda-requirements.txt" that has the high-level
requirements for installation
2. create a temporary environment, install conda-requirements into it.
3. Dump conda list --explicit to a conda-spec.<platform>.txt

and we run (1), (2), (3) on OSX and also on a centos box (allowing
development on osx and on linux).

This improves build times and also ensures complete clarity on what ends up
installed in your environment. Unfortunately, we keep hitting issues where
packages move around and the previous references are no longer valid.

So I'm not sure what to do here - is it intentional that conda specs should
not continue to work? Would it be possible to have conda.anaconda.org keep
packages forever once uploaded?

I'm not 100% clear on why older packages continue to get removed either -
but my understanding it's a combination of the changes conda-forge and
bioconda have been making for the newest version of conda and how
conda.anaconda.org handles package changes.

Thank you in advance for your consideration and advice,

Jeff

--

Jeff Tratner
@jtratner on many things