[conda] meta.yaml versus setup.py

[Chris Withers]

Hi All,

I'm trying to build my first conda-only package and I'm confused between
the apparent duplication between meta.yaml and setup.py.

Here's my current meta.yaml:
https://gist.github.com/cjw296/481b155fa63e3d48241add1946f389d5

Everything that would be in setup.py appears to be in here, and yet if I
try a "PROJECT_VERSION=dev conda build ." I get:

...
Executing transaction: ...working... done
Copying /Users/chris/vcs/git/picky-conda to
/Users/chris/anaconda2/conda-bld/picky-conda_1523432431361/work
source tree in:
/Users/chris/anaconda2/conda-bld/picky-conda_1523432431361/work
python: can't open file 'setup.py': [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/Users/chris/anaconda2/bin/conda-build", line 11, in <module>
...
  File
"/Users/chris/anaconda2/lib/python2.7/site-packages/conda_build/utils.py",
line 280, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e',
'/Users/chris/anaconda2/conda-bld/picky-conda_1523432431361/work/conda_build.sh']'
returned non-zero exit status 2

Why does setup.py need to exist, and what gets used from it in
comparison to that in meta.yaml? (package name? version? entry_points?).

I also feel like I might be wrong in my expectation that "conda build ."
is the same as the "pip install -e .[build,test]" I'd do in a
pip-focused package. What's the conda equivalent?
(I need my entry point created so I can test it and I test my build/test
requirements installed so I can run pytest/coverage/etc).

If I put a minimal setup.py back in, as shown in the gist above, the
"PROJECT_VERSION=dev conda build ." eventually craps out with:

Executing transaction: ...working... done
+ picky -h
Traceback (most recent call last):
  File
"/Users/chris/anaconda2/conda-bld/picky-conda_1523433369030/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/picky",
line 6, in <module>
    from picky.main import main
ModuleNotFoundError: No module named 'picky'
Tests failed for picky-conda-dev-py_0.tar.bz2 - moving package to
/Users/chris/anaconda2/conda-bld/broken
WARNING:conda_build.build:Tests failed for picky-conda-dev-py_0.tar.bz2
- moving package to /Users/chris/anaconda2/conda-bld/broken
removing: picky-conda-dev-py_0.tar.bz2
TESTS FAILED: picky-conda-dev-py_0.tar.bz2

What am I doing wrong?

cheers,

Chris

[Chris Withers]

Generally setup.py is necessary to support both PyPI packages of your
library and conda packages.  For development recipes, I actually
recommend putting most of the information into setup.py and/or
setup.cfg, and then parsing that data in meta.yaml.  Here's an example
https://github.com/gerrymandr/adjacency-graphs/blob/master/conda.recipe/meta.yaml

Interesting, in this case it's a tool which is only going to usable in a
conda environment...

"conda build ." is definitely not the same as "pip install -e ." - there
is no conda equivalent, because there really doesn't need to be one.

So, if I have a clone of a git repo and an activate conda environment,
how do I "develop" my package? (ie: get my testing-only libraries
installed, get my entry points built for the code in the git clone).

I believe you may be asking for features to create a build/test env,
then have conda-build run a develop install for you.  I also think
you've previously asked this question in the thread at
https://groups.google.com/a/continuum.io/forum/#!topic/conda/yov5Ua1_NLI,

Almost ;-) that one was actually for an application that assembles a
bunch of packages. This one is for developing and testing a specific
utility for conda.
<snip>

I read that as "you want conda develop, but that's broken, so don't use
it". Okay, but what *should* I do/use right now?

cheers,

Chris

[Chris Withers]

Post by Chris Withers
pip install -e .
is my preferred method

But won't that pull in library dependencies (especially binary ones, say
numpy, etc) from pypi instead on anaconda.org?

cheers,

Chris

[Chris Barker]

Post by Chris Withers

Post by Chris Withers
pip install -e .
is my preferred method

But won't that pull in library dependencies (especially binary ones, say
numpy, etc) from pypi instead on anaconda.org?

And now you know why I wish pip install -e DID NOT pull in dependencies for
you by default :-)

so do:

pip install --no-deps -e .

and before that, do something like:

conda install --file conda_requirements.txt

or

conda create -n name_of_env --file conda_requirements.txt

or build and environment with a environment.yaml file.

in fact, if your conda dependencies are all build correctly, then pipi will
find that they are there, and you don't need the --no-deps

I think the key problem here is that conda was developed for, and is widely
used for Python packages, so we all expect it to act like an extension to
pip.....

But in fact, conda is designed to manage any kind of package (importantly
Python itself), so while it is pretty pip friendly, it really is orthogonal
to pip.

In fact, it's pip that is not friendly to other package managers :-)

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Paul Hobson]

Oops. I accidentally took part of the discussion offline.

I asked:
Great example, Michael. Is load_setup_py_data always in the namespace of
meta.yml?

Michael responded:
Yes, it is one of the jinja2 functions provided. If you'd like to see all
about what goes into the jinja2 context, start at https://github.com/conda/
conda-build/blob/master/conda_build/jinja_context.py#L492-L529 and explore
backwards. Thanks for the great info, Michael!
-paul

Generally setup.py is necessary to support both PyPI packages of your
library and conda packages. For development recipes, I actually recommend
putting most of the information into setup.py and/or setup.cfg, and then
https://github.com/gerrymandr/adjacency-graphs/blob/master/
conda.recipe/meta.yaml
You may find the slides that I presented at AnacondaCon this week to be
helpful: https://docs.google.com/presentation/d/
1E3fKfnUNuvgqWW5QL1D6Z59QLM7XGvYfkvl4zTVMXZE/edit#slide=id.p3
"conda build ." is definitely not the same as "pip install -e ." - there
is no conda equivalent, because there really doesn't need to be one.
"conda build ." makes a package, whereas "pip install -e ." makes a
development link from your source location on your hard drive to the python
installation. They are completely separate concerns. Packages are for
redistribution, development links are for local development.
I believe you may be asking for features to create a build/test env, then
have conda-build run a develop install for you. I also think you've
previously asked this question in the thread at
https://groups.google.com/a/continuum.io/forum/#!topic/conda/yov5Ua1_NLI,
https://github.com/conda/conda-build/issues/1992#issuecomment-322588270
https://github.com/conda/conda-build/issues/2633
https://github.com/conda/conda-build/issues/1677
https://github.com/conda/conda-build/issues/2791

Post by Chris Withers
Hi All,
I'm trying to build my first conda-only package and I'm confused between
the apparent duplication between meta.yaml and setup.py.
Here's my current meta.yaml: https://gist.github.com/cjw296
/481b155fa63e3d48241add1946f389d5
Everything that would be in setup.py appears to be in here, and yet if I
...
Executing transaction: ...working... done
Copying /Users/chris/vcs/git/picky-conda to
/Users/chris/anaconda2/conda-bld/picky-conda_1523432431361/work
source tree in: /Users/chris/anaconda2/conda-b
ld/picky-conda_1523432431361/work
python: can't open file 'setup.py': [Errno 2] No such file or directory
File "/Users/chris/anaconda2/bin/conda-build", line 11, in <module>
...
File "/Users/chris/anaconda2/lib/python2.7/site-packages/conda_build/utils.py",
line 280, in _func_defaulting_env_to_os_environ
raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e',
'/Users/chris/anaconda2/conda-bld/picky-conda_1523432431361/work/conda_build.sh']'
returned non-zero exit status 2
Why does setup.py need to exist, and what gets used from it in comparison
to that in meta.yaml? (package name? version? entry_points?).
I also feel like I might be wrong in my expectation that "conda build ."
is the same as the "pip install -e .[build,test]" I'd do in a pip-focused
package. What's the conda equivalent?
(I need my entry point created so I can test it and I test my build/test
requirements installed so I can run pytest/coverage/etc).
If I put a minimal setup.py back in, as shown in the gist above, the
Executing transaction: ...working... done
+ picky -h
File "/Users/chris/anaconda2/conda-bld/picky-conda_1523433369030/
_test_env_placehold_placehold_placehold_placehold_placehold_
placehold_placehold_placehold_placehold_placehold_placehold_
placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/picky",
line 6, in <module>
from picky.main import main
ModuleNotFoundError: No module named 'picky'
Tests failed for picky-conda-dev-py_0.tar.bz2 - moving package to
/Users/chris/anaconda2/conda-bld/broken
WARNING:conda_build.build:Tests failed for picky-conda-dev-py_0.tar.bz2
- moving package to /Users/chris/anaconda2/conda-bld/broken
removing: picky-conda-dev-py_0.tar.bz2
TESTS FAILED: picky-conda-dev-py_0.tar.bz2
What am I doing wrong?
cheers,
Chris
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit https://groups.google.com/a/co
ntinuum.io/d/msgid/conda/3ea256dc-8b23-c4ca-5233-78593aa4ad3
4%40withers.org.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CAOQN5OgQm%3DQw6Gjt6ZwVm7q73FoXz41vF11Eh6
KJQup0dJ7N4g%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CAOQN5OgQm%3DQw6Gjt6ZwVm7q73FoXz41vF11Eh6KJQup0dJ7N4g%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Chris Withers]

Hi Michael,

Generally setup.py is necessary to support both PyPI packages of your
library and conda packages.  For development recipes, I actually
recommend putting most of the information into setup.py and/or
setup.cfg, and then parsing that data in meta.yaml.  Here's an example
https://github.com/gerrymandr/adjacency-graphs/blob/master/conda.recipe/meta.yaml
You may find the slides that I presented at AnacondaCon this week to be
https://docs.google.com/presentation/d/1E3fKfnUNuvgqWW5QL1D6Z59QLM7XGvYfkvl4zTVMXZE/edit#slide=id.p3

Great presentation! Really helpful :-)

Off the back of that, I updated my setup.py and meta.yaml, I've put them
on this gist:

https://gist.github.com/cjw296/481b155fa63e3d48241add1946f389d5

However, if I do:

conda env create picky-conda python=3.6
conda activate picky-conda
pip install -e .[test,build]

When trying to do a build I get:

$ conda build .
Error: package/name missing in:
u'/Users/chris/vcs/git/picky-conda/meta.yaml'

How can I debug that?

cheers,

Chris

[Chris Withers]

Post by Chris Withers
Hi Michael,

Generally setup.py is necessary to support both PyPI packages of your
library and conda packages.  For development recipes, I actually
recommend putting most of the information into setup.py and/or
setup.cfg, and then parsing that data in meta.yaml.  Here's an example
https://github.com/gerrymandr/adjacency-graphs/blob/master/conda.recipe/meta.yaml
You may find the slides that I presented at AnacondaCon this week to
https://docs.google.com/presentation/d/1E3fKfnUNuvgqWW5QL1D6Z59QLM7XGvYfkvl4zTVMXZE/edit#slide=id.p3

Great presentation! Really helpful :-)
Off the back of that, I updated my setup.py and meta.yaml, I've put them
https://gist.github.com/cjw296/481b155fa63e3d48241add1946f389d5
conda env create picky-conda python=3.6
conda activate picky-conda
pip install -e .[test,build]
$ conda build .
u'/Users/chris/vcs/git/picky-conda/meta.yaml'
How can I debug that?

Forgot to include:

$ conda list | grep build
conda-build 3.8.0 py27_0

Chris