[conda] How to constrain the version of packages using the new pinning syntax?

[Bryan W. Weber]

Hello,

I'm building a package that I know doesn't work for some versions of my
dependencies (for instance, I need NumPy >= 1.8.1). I'm using the new
pinning syntax, so something like

...
requirements:
build:
- python {{ python }}
- numpy {{ numpy }}
run:
- python
- {{ pin_compatible(numpy, min_pin='x.x', max_pin='x.x') }}
...

How can I restrict the versions of, for example, NumPy to be greater than
1.8.1 with this syntax? I tried

...
- numpy >=1.8.1,{{ numpy }}
...

but if I run this with the print_yamls function defined in the blog post
announcing conda-build 3.0
(https://www.anaconda.com/blog/developer-blog/package-better-conda-build-3/?lang=en-us),
I get an error

print_yamls('cantera/meta.yaml', numpy='1.13', python='3.6')

... Traceback ...

DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-
64: {'numpy >=1.8.1,1.13'}

If I put a * at the end of the dependency line

...
- numpy >=1.8.1,{{ numpy }}*
...

then the print_yamls function gives the expected output. Is this the
correct syntax? How can I then express more complicated dependencies? For
instance, I need Python 2.7 or Python >=3.3. If I try

...
- python >=2.7,<3|>=3.3,{{ python }}*
...

the print_yamls function finds Python 2.7 as the requirement, no matter
what I put for the python argument to the print_yamls function. This
happens with or without the * at the end of the line.

My overall question then is, what is the correct syntax to use the new
pinning syntax and limit the compatible versions of various packages?

Thank you!
Bryan

[Bryan W. Weber]

Michael,

Thanks for the quick reply, and apologies for my delay. What you said makes
sense, so I'm trying to implement that now. As one additional wrinkle, I'm
also trying to do this with subpackages. However, I'm experiencing some
unexpected behavior with the subpackages, so I'll ask the generic questions
first, and see if the answers fix the specific questions I have about my
project.

Given the following meta.yaml:

package:
name: subpackage_test
version: 1.0.0

outputs:
- name: subtest-1
requirements:
build:
- {{ compiler('c') }}
- {{ compiler('cxx') }}
script: build_subtest.sh
- name: subpackage_test
requirements:
build:
- numpy
# run:
# - {{ pin_compatible('numpy') }}

I have a few questions about this.

1) The run/pin_compatible is commented out because when I run conda build
on this recipe, it returns an error
Attempting to finalize metadata for subpackage_test
INFO:conda_build.metadata:Attempting to finalize metadata for
subpackage_test
Could not get compatibility information for numpy package. Is it one of
your build dependencies?

Apparently, having the build requirement in the same subpackage is not
sufficient?

2) Given the following build_subtest.sh script:
echo "CC is"
echo $CC
echo "CXX is"
echo $CXX

The output is (truncated):

INFO:conda_build.build:Packaging subtest-1


The following NEW packages will be INSTALLED:


binutils_impl_linux-64: 2.28.1-h04c84fa_2
binutils_linux-64: 7.2.0-hc67e822_15
gcc_impl_linux-64: 7.2.0-hc5ce805_2
gcc_linux-64: 7.2.0-haf1f6fa_15
gxx_impl_linux-64: 7.2.0-hd3faf3d_2
gxx_linux-64: 7.2.0-h2e50c1c_15
libstdcxx-ng: 7.2.0-h7a57d05_2


CC is


CXX is


number of files: 0

So the packages are all installed into an environment, but the environment
is not activated for the packaging step, resulting in the environment
variables not being present. Is this the expected behavior? In addition, if
any top-level build requirements are specified, those packages are
installed into a new environment as expected. However, before the packaging
steps, that environment is removed and the packaging step creates an
environment with the same name in the same location. Is that also expected?

3) The build/packaging step for a subpackage named the same as the
top-level package doesn't install an environment at all. This is the output
from closer to the beginning:

BUILD START: ['subpackage_test-1.0.0-he8bb03b_0.tar.bz2',
'subtest-1-1.0.0-h9767510_0.tar.bz2']
source tree in: /home/bryan/miniconda3/conda-bld/
subpackage_test_1511811557450/work
Packaging subpackage_test
INFO:conda_build.build:Packaging subpackage_test
Packaging subpackage_test-1.0.0-he8bb03b_0
INFO:conda_build.build:Packaging subpackage_test-1.0.0-he8bb03b_0
number of files: 0
Fixing permissions
Fixing permissions
updating: subpackage_test-1.0.0-he8bb03b_0.tar.gz

4) On an unrelated note, the following meta.yaml

package:
name: test
version: 1.0.0


outputs:
- name: subtest-1
requirements:
build:
- {{ compiler('c') }}
- {{ compiler('cxx') }}
- name: test

produces the following output:

Attempting to finalize metadata for subtest-1
INFO:conda_build.metadata:Attempting to finalize metadata for subtest-1
Attempting to finalize metadata for test
INFO:conda_build.metadata:Attempting to finalize metadata for test
BUILD START: ['subtest-1-1.0.0-hde4badb_0.tar.bz2',
'test-1.0.0-hcbb2ae2_0.tar.bz2']
source tree in: /home/bryan/miniconda3/conda-bld/test_1511812918250/work
Packaging subtest-1
INFO:conda_build.build:Packaging subtest-1


The following NEW packages will be INSTALLED:


binutils_impl_linux-64: 2.28.1-h04c84fa_2
binutils_linux-64: 7.2.0-hc67e822_15
gcc_impl_linux-64: 7.2.0-hc5ce805_2
gcc_linux-64: 7.2.0-haf1f6fa_15
gxx_impl_linux-64: 7.2.0-hd3faf3d_2
gxx_linux-64: 7.2.0-h2e50c1c_15
libstdcxx-ng: 7.2.0-h7a57d05_2


Packaging subtest-1-1.0.0-hde4badb_0
INFO:conda_build.build:Packaging subtest-1-1.0.0-hde4badb_0
number of files: 0
Fixing permissions
Fixing permissions
updating: subtest-1-1.0.0-hde4badb_0.tar.bz2
Packaging test
INFO:conda_build.build:Packaging test
number of files: 3433
Fixing permissions
Making absolute symlink x86_64-conda_cos6-linux-gnu/sysroot/lib/libm.so.6 ->
libm-2.12.2.so relative
Making absolute symlink libexec/gcc/x86_64-conda_cos6-linux-gnu/7.2.0/
liblto_plugin.so.0 -> liblto_plugin.so.0.0.0 relative
Making absolute symlink x86_64-conda_cos6-linux-gnu/lib/libstdc++.so ->
libstdc++.so.6.0.24 relative
...
Warning: rpath /home/msarahan/miniconda2/conda-bld/compilers_linux-64
_1507259624353/
_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
/lib is outside prefix /home/bryan/miniconda3/conda-bld/test_1511812918250/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
(removing it)

and so on for many more files. Note the msarahan in the output. This only
occurs with the particular combination of package dependencies and names
that are in the meta.yaml file (i.e., the package must be named test, and
the subpackages must be named subtest-1 and test, and the compilers must be
installed). Is this an easter egg? :-)

Thank you!
Bryan

It is best to think of it as the build section determining your lower
bound. Set a value for numpy in conda_build_config.yaml accordingly. For
https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml#L95-L96.
This is because we had issues compiling scipy against the older numpy on
windows.
The {{ numpy }} isn't strictly necessary. Conda-build will try to match
up names with entries in conda_build_config.yaml for the build and host
sections. I call this "implicit pinning." This does not apply to the run
or test sections.
OK, with the baseline numpy version set, there's two ways to do the numpy
1. You use the numpy C api, so you need to be more conservative about
pinning.
{{ pin_compatible('numpy') }}
This becomes >=(baseline ver installed at build time),<(next major ver)
You only need to change min_pin and max_pin if you want to make min_pin
*less specific* or max_pin *more specific*.
2. You only use numpy via python, so you don't need to pin as
conservatively
numpy >={{ numpy }}
3. You want the dynamic lower bound, but a hardcoded upper bound
{{ pin_compatible('numpy', upper_bound=(hardcoded upper version) }}
4. You need both lower and upper bounds customizable to something other
Don't use pin_compatible. It isn't buying you anything.
numpy >=(hardcoded lower),<(hardcoded upper)

Post by Bryan W. Weber
Hello,
I'm building a package that I know doesn't work for some versions of my
dependencies (for instance, I need NumPy >= 1.8.1). I'm using the new
pinning syntax, so something like
...
- python {{ python }}
- numpy {{ numpy }}
- python
- {{ pin_compatible(numpy, min_pin='x.x', max_pin='x.x') }}
...
How can I restrict the versions of, for example, NumPy to be greater than
1.8.1 with this syntax? I tried
...
- numpy >=1.8.1,{{ numpy }}
...
but if I run this with the print_yamls function defined in the blog post
announcing conda-build 3.0 (
https://www.anaconda.com/blog/developer-blog/package-better-conda-build-3/?lang=en-us),
I get an error

print_yamls('cantera/meta.yaml', numpy='1.13', python='3.6')

... Traceback ...
DependencyNeedsBuildingError: Unsatisfiable dependencies for platform
linux-64: {'numpy >=1.8.1,1.13'}
If I put a * at the end of the dependency line
...
- numpy >=1.8.1,{{ numpy }}*
...
then the print_yamls function gives the expected output. Is this the
correct syntax? How can I then express more complicated dependencies? For
instance, I need Python 2.7 or Python >=3.3. If I try
...
- python >=2.7,<3|>=3.3,{{ python }}*
...
the print_yamls function finds Python 2.7 as the requirement, no matter
what I put for the python argument to the print_yamls function. This
happens with or without the * at the end of the line.
My overall question then is, what is the correct syntax to use the new
pinning syntax and limit the compatible versions of various packages?
Thank you!
Bryan
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/f5b687d3-ed5d-4b27-a8d6-4bcab33a0601%40continuum.io
<https://groups.google.com/a/continuum.io/d/msgid/conda/f5b687d3-ed5d-4b27-a8d6-4bcab33a0601%40continuum.io?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

[Bryan W. Weber]

Thank you again for your quick help!

1) Thanks! Should I post an issue on GH?

2) OK, it's easy enough to activate the environment. Can I rely on the
environment being the same name as the top-level build environment, or at
least that the PREFIX will be the same? The reason I'm asking is because
I'm using the full path to the compiler in my build configuration files,
and I want to make sure I don't need to change that on each packaging step.

3) and 4) OK, if I install conda build from the master branch, it builds
everything. Hooray! Small hiccup: I have a conda_build_config.yaml with
three versions of Python (2.7, 3.5, and 3.6) and two versions of Numpy
(1.12 and 1.13). It seems like the top-level build is being run for each of
the builds in the product of those variables. Is that expected? The project
I'm trying to build has a common shared library, and the Cython interface
is linked to that. Only the Cython interface relies on Numpy, so I was
hoping to build the common library once and then build the Cython interface
several times, each time linking to the common library, and with the state
of the directory restored to the post-top-level-build state before each
packaging step. Is it possible to do this? I'm using
commit https://github.com/conda/conda-build/commit/02c6350b21d449c7dd501dcb5dc6ae8d8c23645d
for conda-build.

I guess this is what pin_subpackage is for? The recipe I'm working on is
here: https://github.com/Cantera/conda-recipes/tree/update_conda_build/cantera.
In my case, should I have the cantera subpackage output require the
libcantera output during its build phase, and link against the libcantera
installed in the packaging environment? I think I need to avoid doing this
because I don't have a way to use our build system (SCons, FWIW) to build
just the Python interface, it will always check for the common library in
the build tree and, not finding it, build the whole library again. Hence,
in my recipe, I'm trying to build the common library with the top-level
build.sh, and build/install the Python/Cython interfaces in the cantera
subpackage. The libcantera subpackage then only handles installing the
library to the PREFIX, and isn't actually required to run the Python
interface, but I thought it would be a "nice-to-have" if our users wanted
to install that (we have a C++ interface to the library as well, that would
be covered by the libcantera package).

Thank you!
Bryan

1) seems like a bug. I'll try to figure it out. What you have there
should work, but it might want numpy in the build or host section in the
top-level recipe, even if they're not used for anything there.
2) the environments created for each packaging step are completely
independent from the top-level environments, and yes, they do replace the
https://github.com/conda/conda-build/blob/master/conda_build/build.py#L1166
The environments are not activated during execution of the packaging
scripts. Because the packaging scripts are so open-ended, it's not really
possible to append activation commands to them. They can be any kind of
file. We do put the build prefix on PATH, but doing all of the arbitrary
activation tasks is not done.
You can/should add "source activate $PREFIX" to the top of your packaging
script if you need it.
3) fixed in https://github.com/conda/conda-build/pull/2511
4) I believe this is also fixed by
https://github.com/conda/conda-build/pull/2511, but may not be. Having
outputs named the same as the top-level name implies the old behavior. It
should probably be explicitly broken when any other outputs are present.
It's basically a hack to make the old way into the new way - to create an
output when there is no explicit output specified.

Post by Bryan W. Weber
Michael,
Thanks for the quick reply, and apologies for my delay. What you said
makes sense, so I'm trying to implement that now. As one additional
wrinkle, I'm also trying to do this with subpackages. However, I'm
experiencing some unexpected behavior with the subpackages, so I'll ask the
generic questions first, and see if the answers fix the specific questions
I have about my project.
name: subpackage_test
version: 1.0.0
- name: subtest-1
- {{ compiler('c') }}
- {{ compiler('cxx') }}
script: build_subtest.sh
- name: subpackage_test
- numpy
# - {{ pin_compatible('numpy') }}
I have a few questions about this.
1) The run/pin_compatible is commented out because when I run conda build
on this recipe, it returns an error
Attempting to finalize metadata for subpackage_test
INFO:conda_build.metadata:Attempting to finalize metadata for
subpackage_test
Could not get compatibility information for numpy package. Is it one of
your build dependencies?
Apparently, having the build requirement in the same subpackage is not
sufficient?
echo "CC is"
echo $CC
echo "CXX is"
echo $CXX
INFO:conda_build.build:Packaging subtest-1
binutils_impl_linux-64: 2.28.1-h04c84fa_2
binutils_linux-64: 7.2.0-hc67e822_15
gcc_impl_linux-64: 7.2.0-hc5ce805_2
gcc_linux-64: 7.2.0-haf1f6fa_15
gxx_impl_linux-64: 7.2.0-hd3faf3d_2
gxx_linux-64: 7.2.0-h2e50c1c_15
libstdcxx-ng: 7.2.0-h7a57d05_2
CC is
CXX is
number of files: 0
So the packages are all installed into an environment, but the
environment is not activated for the packaging step, resulting in the
environment variables not being present. Is this the expected behavior? In
addition, if any top-level build requirements are specified, those packages
are installed into a new environment as expected. However, before the
packaging steps, that environment is removed and the packaging step creates
an environment with the same name in the same location. Is that also
expected?
3) The build/packaging step for a subpackage named the same as the
top-level package doesn't install an environment at all. This is the output
BUILD START: ['subpackage_test-1.0.0-he8bb03b_0.tar.bz2',
'subtest-1-1.0.0-h9767510_0.tar.bz2']
source tree in: /home/bryan/miniconda3/conda-bld/
subpackage_test_1511811557450/work
Packaging subpackage_test
INFO:conda_build.build:Packaging subpackage_test
Packaging subpackage_test-1.0.0-he8bb03b_0
INFO:conda_build.build:Packaging subpackage_test-1.0.0-he8bb03b_0
number of files: 0
Fixing permissions
Fixing permissions
updating: subpackage_test-1.0.0-he8bb03b_0.tar.gz
4) On an unrelated note, the following meta.yaml
name: test
version: 1.0.0
- name: subtest-1
- {{ compiler('c') }}
- {{ compiler('cxx') }}
- name: test
Attempting to finalize metadata for subtest-1
INFO:conda_build.metadata:Attempting to finalize metadata for subtest-1
Attempting to finalize metadata for test
INFO:conda_build.metadata:Attempting to finalize metadata for test
BUILD START: ['subtest-1-1.0.0-hde4badb_0.tar.bz2',
'test-1.0.0-hcbb2ae2_0.tar.bz2']
source tree in: /home/bryan/miniconda3/conda-bld/test_1511812918250/work
Packaging subtest-1
INFO:conda_build.build:Packaging subtest-1
binutils_impl_linux-64: 2.28.1-h04c84fa_2
binutils_linux-64: 7.2.0-hc67e822_15
gcc_impl_linux-64: 7.2.0-hc5ce805_2
gcc_linux-64: 7.2.0-haf1f6fa_15
gxx_impl_linux-64: 7.2.0-hd3faf3d_2
gxx_linux-64: 7.2.0-h2e50c1c_15
libstdcxx-ng: 7.2.0-h7a57d05_2
Packaging subtest-1-1.0.0-hde4badb_0
INFO:conda_build.build:Packaging subtest-1-1.0.0-hde4badb_0
number of files: 0
Fixing permissions
Fixing permissions
updating: subtest-1-1.0.0-hde4badb_0.tar.bz2
Packaging test
INFO:conda_build.build:Packaging test
number of files: 3433
Fixing permissions
Making absolute symlink x86_64-conda_cos6-linux-gnu/sysroot/lib/libm.so.6
-> libm-2.12.2.so relative
Making absolute symlink libexec/gcc/x86_64-conda_cos6-linux-gnu/7.2.0/
liblto_plugin.so.0 -> liblto_plugin.so.0.0.0 relative
Making absolute symlink x86_64-conda_cos6-linux-gnu/lib/libstdc++.so ->
libstdc++.so.6.0.24 relative
...
Warning: rpath /home/msarahan/miniconda2/conda-bld/compilers_linux-64
_1507259624353/
_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
/lib is outside prefix /home/bryan/miniconda3/conda-bld/
test_1511812918250/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
(removing it)
and so on for many more files. Note the msarahan in the output. This only
occurs with the particular combination of package dependencies and names
that are in the meta.yaml file (i.e., the package must be named test, and
the subpackages must be named subtest-1 and test, and the compilers must be
installed). Is this an easter egg? :-)
Thank you!
Bryan

It is best to think of it as the build section determining your lower
bound. Set a value for numpy in conda_build_config.yaml accordingly. For
https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml#L95-L96.
This is because we had issues compiling scipy against the older numpy on
windows.
The {{ numpy }} isn't strictly necessary. Conda-build will try to match
up names with entries in conda_build_config.yaml for the build and host
sections. I call this "implicit pinning." This does not apply to the run
or test sections.
OK, with the baseline numpy version set, there's two ways to do the
1. You use the numpy C api, so you need to be more conservative about
pinning.
{{ pin_compatible('numpy') }}
This becomes >=(baseline ver installed at build time),<(next major ver)
You only need to change min_pin and max_pin if you want to make min_pin
*less specific* or max_pin *more specific*.
2. You only use numpy via python, so you don't need to pin as
conservatively
numpy >={{ numpy }}
3. You want the dynamic lower bound, but a hardcoded upper bound
{{ pin_compatible('numpy', upper_bound=(hardcoded upper version) }}
4. You need both lower and upper bounds customizable to something other
Don't use pin_compatible. It isn't buying you anything.
numpy >=(hardcoded lower),<(hardcoded upper)

Post by Bryan W. Weber
Hello,
I'm building a package that I know doesn't work for some versions of my
dependencies (for instance, I need NumPy >= 1.8.1). I'm using the new
pinning syntax, so something like
...
- python {{ python }}
- numpy {{ numpy }}
- python
- {{ pin_compatible(numpy, min_pin='x.x', max_pin='x.x') }}
...
How can I restrict the versions of, for example, NumPy to be greater
than 1.8.1 with this syntax? I tried
...
- numpy >=1.8.1,{{ numpy }}
...
but if I run this with the print_yamls function defined in the blog
post announcing conda-build 3.0 (
https://www.anaconda.com/blog/developer-blog/package-better-conda-build-3/?lang=en-us),
I get an error

print_yamls('cantera/meta.yaml', numpy='1.13', python='3.6')

... Traceback ...
DependencyNeedsBuildingError: Unsatisfiable dependencies for platform
linux-64: {'numpy >=1.8.1,1.13'}
If I put a * at the end of the dependency line
...
- numpy >=1.8.1,{{ numpy }}*
...
then the print_yamls function gives the expected output. Is this the
correct syntax? How can I then express more complicated dependencies? For
instance, I need Python 2.7 or Python >=3.3. If I try
...
- python >=2.7,<3|>=3.3,{{ python }}*
...
the print_yamls function finds Python 2.7 as the requirement, no
matter what I put for the python argument to the print_yamls function.
This happens with or without the * at the end of the line.
My overall question then is, what is the correct syntax to use the new
pinning syntax and limit the compatible versions of various packages?
Thank you!
Bryan
--
You received this message because you are subscribed to the Google
Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at
https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/f5b687d3-ed5d-4b27-a8d6-4bcab33a0601%40continuum.io
<https://groups.google.com/a/continuum.io/d/msgid/conda/f5b687d3-ed5d-4b27-a8d6-4bcab33a0601%40continuum.io?utm_medium=email&utm_source=footer>
.
For more options, visit
https://groups.google.com/a/continuum.io/d/optout.

--

You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/d9b6c891-0e08-4777-b648-c9d007fd517b%40continuum.io
<https://groups.google.com/a/continuum.io/d/msgid/conda/d9b6c891-0e08-4777-b648-c9d007fd517b%40continuum.io?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.