[conda] Long term plan for standalone conda installer (conda.install replacement)

[Phil Elson]

Hi all,

Just digging a little bit into some of the changes in the conda codebase,
particularly in regard to the abilities of *conda.install*

At one point the contents of conda.install were vendored out into places
like the miniconda installer (as a bzip). I followed a similar pattern when
I created conda-rpms <https://github.com/scitools-incubator/conda-rpms>,
and to a lesser extent other tools like conda-execute
<https://github.com/conda-tools/conda-execute>. In fact, the conda.install
code is in a lot of places (libconda
<https://github.com/conda/libconda/blob/master/libconda/install.py>,
miniconda <https://conda.io/miniconda.html> and conda-constructor
<https://github.com/conda/constructor/blob/master/constructor/install.py>).
The conda.install code was a little brittle, and extremely sensitive as it
could pretty-much only depend on the standard library (because it was being
used to install python+conda itself!).

The huge downside of all this vendoring is that it makes it really hard to
start adding new functionality to conda distributions. A really great
example of this is noarch python distributions, which are not supported in
most of the tools listed above (with the exception on conda-execute, thanks
to Kale's effort <https://github.com/conda-tools/conda-execute/pull/43> a
few months ago). As it happens, the particular noarch_python issue could be
solved by vendoring a newer version of install.py
<https://github.com/conda/conda/blob/ad741a23f346ef9b4e759fa99503d624d2055103/conda/install.py>
.

The conda codebase appears to be moving away from this ability altogether
(despite the *conda.install* docstring currently saying "low-level code for
extracting, linking and unlinking packages"
<https://github.com/conda/conda/blob/master/conda/install.py#L4>, it
doesn't actually have any low-level functionality in it anymore). My
perception is that conda now heavily depends upon conda (and all of its
dependencies) to install conda binary distributions (tar.bz2 files). So I'm
really interested in the long-term plans regarding the bootstrapping of a
conda installation (e.g. Miniconda).

Today's Miniconda looks like it is using a really old version of the code
that was once conda.install (before noarch was even a thing!). We have
therefore arrived at a situation where the creation of self extracting
tarballs (such as Miniconda.sh, or those created with conda-constructor)
will not be able to install entirely valid conda binary distributions (such
as noarch).

Personally, I have an interest in the ability to install conda
distributions without the need to install conda itself, which seems to
align with the tools I've previously mentioned. So my question really is
whether or not there is a plan regarding this change? What are the plans
for the miniconda installer, or is it just going to stay as it is an not
actually use any of the new features (e.g. noarch packages). If that is the
case, will conda-construtor forever remain unable to install noarch
packages? Similarly, is libconda a Dodo at this point?

Would really appreciate getting any info on the direction of travel so that
I can plan appropriately for use of conda-construtor and development of
conda-rpms.

Thanks in advance,

Phil


P.S. If anybody is interested in conda-rpms, you might like to read
https://rawgit.com/scitools-incubator/conda-rpms/
e8e3287a02e9cef9aa5c8e0bfd7f95d95147edc0/tmp_overview_docs/
scitools-env-workflow.html. Happy to discuss it further in another thread.

[Laura Dreyer]

I would like to bump this discussion.
Are there any thoughts on what to do about the conda installer?

Post by Phil Elson
Hi all,
Just digging a little bit into some of the changes in the conda codebase,
particularly in regard to the abilities of *conda.install*
At one point the contents of conda.install were vendored out into places
like the miniconda installer (as a bzip). I followed a similar pattern when
I created conda-rpms <https://github.com/scitools-incubator/conda-rpms>,
and to a lesser extent other tools like conda-execute
<https://github.com/conda-tools/conda-execute>. In fact, the
conda.install code is in a lot of places (libconda
<https://github.com/conda/libconda/blob/master/libconda/install.py>,
miniconda <https://conda.io/miniconda.html> and conda-constructor
<https://github.com/conda/constructor/blob/master/constructor/install.py>).
The conda.install code was a little brittle, and extremely sensitive as it
could pretty-much only depend on the standard library (because it was being
used to install python+conda itself!).
The huge downside of all this vendoring is that it makes it really hard to
start adding new functionality to conda distributions. A really great
example of this is noarch python distributions, which are not supported in
most of the tools listed above (with the exception on conda-execute, thanks
to Kale's effort <https://github.com/conda-tools/conda-execute/pull/43> a
few months ago). As it happens, the particular noarch_python issue could be
solved by vendoring a newer version of install.py
<https://github.com/conda/conda/blob/ad741a23f346ef9b4e759fa99503d624d2055103/conda/install.py>
.
The conda codebase appears to be moving away from this ability altogether
(despite the *conda.install* docstring currently saying "low-level code
for extracting, linking and unlinking packages"
<https://github.com/conda/conda/blob/master/conda/install.py#L4>, it
doesn't actually have any low-level functionality in it anymore). My
perception is that conda now heavily depends upon conda (and all of its
dependencies) to install conda binary distributions (tar.bz2 files). So I'm
really interested in the long-term plans regarding the bootstrapping of a
conda installation (e.g. Miniconda).
Today's Miniconda looks like it is using a really old version of the code
that was once conda.install (before noarch was even a thing!). We have
therefore arrived at a situation where the creation of self extracting
tarballs (such as Miniconda.sh, or those created with conda-constructor)
will not be able to install entirely valid conda binary distributions (such
as noarch).
Personally, I have an interest in the ability to install conda
distributions without the need to install conda itself, which seems to
align with the tools I've previously mentioned. So my question really is
whether or not there is a plan regarding this change? What are the plans
for the miniconda installer, or is it just going to stay as it is an not
actually use any of the new features (e.g. noarch packages). If that is the
case, will conda-construtor forever remain unable to install noarch
packages? Similarly, is libconda a Dodo at this point?
Would really appreciate getting any info on the direction of travel so
that I can plan appropriately for use of conda-construtor and development
of conda-rpms.
Thanks in advance,
Phil
P.S. If anybody is interested in conda-rpms, you might like to read
https://rawgit.com/scitools-incubator/conda-rpms/e8e3287a02e9cef9aa5c8e0bfd7f95d95147edc0/tmp_overview_docs/scitools-env-workflow.html.
Happy to discuss it further in another thread.