[conda] conda envs and "editable" install of packages?

[Chris Withers]

[now that I've found what's hopefully the correct forum]

Hi All,

What's the conda env equivalent of "pip install -e ."?

"conda env export" certainly gets this wrong and doesn't seem to handle
rounded tripping of editable installs of packages with pip. Where's best
to file a bug for that?


cheers,

Chris

[Chris Barker - NOAA Federal]

Conda develop. Though you can use setup.py develop or pip install -e inside
a conda environment too — and it mostly works.

https://conda.io/docs/commands/build/conda-develop.html

"conda env export" certainly gets this wrong and doesn't seem to handle
rounded tripping of editable installs of packages with pip.


The very nature of conda makes this next to impossible— and certainly not
robust — cause an editable install is not a package, it’s links to on-disk
code. By definition that can’t be reproduced anywhere that same code isn’t
there at the same Path.

Trying to reproduce an environment with editable installs doesn’t really
make sense.

What I do is reproduce the development environment without the “live” code,
then install with editable mode in a separate step.

-CHB


Where's best to file a bug for that?


cheers,

Chris
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to conda+***@continuum.io.
To post to this group, send email to ***@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/4a1755e0-7ffc-9520-625d-d43ddad1c531%40withers.org
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Chris Withers]

Post by Chris Barker - NOAA Federal
Conda develop. Though you can use setup.py develop or pip install -e
inside a conda environment too — and it mostly works.
https://conda.io/docs/commands/build/conda-develop.html

Thanks! I've used "pip install -e ." inside a conda env, mainly to get
console entry points and the like built, does "conda develop" do the
same? Is either "better" in a conda env for any particular reason?

Post by Chris Barker - NOAA Federal

Post by Chris Withers
"conda env export" certainly gets this wrong and doesn't seem to
handle rounded tripping of editable installs of packages with pip.

The very nature of conda makes this next to impossible— and certainly
not robust — cause an editable install is not a package, it’s links to
on-disk code. By definition that can’t be reproduced anywhere that same
code isn’t there at the same Path.

Hmm, pip freeze manages this "ok": figures out git url if there is one
or, iirc, will just use the path on disk, which can be fine for
reproducing things where that path is, say, to a shared filesystem being
used as a local repo.

Post by Chris Barker - NOAA Federal
Trying to reproduce an environment with editable installs doesn’t really
make sense.
What I do is reproduce the development environment without the “live”
code, then install with editable mode in a separate step.

Sounds pragmatic and sensible, but if conda doesn't want to do the same
things that pip freeze does with "develop" or "editable" install,
shouldn't it either filter them out or warn/error?

cheers,

Chris

[Chris Barker]

Post by Chris Barker - NOAA Federal
Conda develop. Though you can use setup.py develop or pip install -e

inside a conda environment too — and it mostly works.
https://conda.io/docs/commands/build/conda-develop.html

Thanks! I've used "pip install -e ." inside a conda env, mainly to get
console entry points and the like built, does "conda develop" do the same?
Is either "better" in a conda env for any particular reason?

in theory, conda develop should handle shared libraries better. I say in
theory, 'cause I use a hacky work-around form before conda develop existed,
and since that works, I haven't tried it.

and maybe exporting the environment will work better?

Post by Chris Barker - NOAA Federal
"conda env export" certainly gets this wrong and doesn't seem to handle

Post by Chris Withers
rounded tripping of editable installs of packages with pip.

The very nature of conda makes this next to impossible— and certainly not
robust — cause an editable install is not a package, it’s links to on-disk
code. By definition that can’t be reproduced anywhere that same code isn’t
there at the same Path.

Hmm, pip freeze manages this "ok": figures out git url if there is one or,
iirc, will just use the path on disk, which can be fine for reproducing
things where that path is, say, to a shared filesystem being used as a
local repo.

I'm actually a little surprised, I would expect conda to punt to pip for
this anyway.

Post by Chris Barker - NOAA Federal

Sounds pragmatic and sensible, but if conda doesn't want to do the same

things that pip freeze does with "develop" or "editable" install, shouldn't
it either filter them out or warn/error?

yeah, putting something that will never work into an environment file would
be a bug. Seems worth reporting.

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Barker]

Conda develop is painful. I highly doubt that it does anything useful for
anyone right now. It is something I want to rewrite into something useful,
but I advise using ``python setup.py develop`` or ``pip install -e .`` from
https://github.com/conda/conda-build/issues/1992#issuecomment-322588270
https://github.com/conda/conda-build/issues/2633
https://github.com/conda/conda-build/issues/1677
https://github.com/conda/conda-build/issues/2791

Thanks for the warning, but to bad.

FWIW, I don't think I want develop mode to create an environment for me. I
don't even want it to handle dependencies.

In fact, I really hate that setup.py develop and pip install -e try to
install deps for me by default.

Having pip install auto install deps is great for the usual "Install this
third party package" use case, but develop mode is for, well, developing,
and when you are developing your code, you really should be thinking about
and managing your deps yourself.

What I do want conda develop to do is handle shared libs properly -- which
pip install does not do.

I'm sure there's an old issue about this somewhere, but I think the issue
is that conda re-writes shared lib headers so that they will be properly
found in the installed environment. but if you setup.py develop, your
shared libs don't get re-written, and you end up picking up system libs,
rather than conda ones.

(all this on OS-X -- maybe it's o non-issue on other platforms)

-CHB



and more...

Post by Chris Barker

Post by Chris Barker - NOAA Federal
Conda develop. Though you can use setup.py develop or pip install -e

inside a conda environment too — and it mostly works.
https://conda.io/docs/commands/build/conda-develop.html

Thanks! I've used "pip install -e ." inside a conda env, mainly to get
console entry points and the like built, does "conda develop" do the same?
Is either "better" in a conda env for any particular reason?

in theory, conda develop should handle shared libraries better. I say in
theory, 'cause I use a hacky work-around form before conda develop existed,
and since that works, I haven't tried it.
and maybe exporting the environment will work better?

Post by Chris Barker - NOAA Federal
"conda env export" certainly gets this wrong and doesn't seem to handle

Post by Chris Withers
rounded tripping of editable installs of packages with pip.

The very nature of conda makes this next to impossible— and certainly
not robust — cause an editable install is not a package, it’s links to
on-disk code. By definition that can’t be reproduced anywhere that same
code isn’t there at the same Path.

Hmm, pip freeze manages this "ok": figures out git url if there is one
or, iirc, will just use the path on disk, which can be fine for reproducing
things where that path is, say, to a shared filesystem being used as a
local repo.

I'm actually a little surprised, I would expect conda to punt to pip for
this anyway.

Post by Chris Barker - NOAA Federal

Sounds pragmatic and sensible, but if conda doesn't want to do the same

things that pip freeze does with "develop" or "editable" install, shouldn't
it either filter them out or warn/error?

yeah, putting something that will never work into an environment file
would be a bug. Seems worth reporting.
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE
<https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g>
[phone] fax
Seattle, WA 98115 [phone] main reception
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit https://groups.google.com/a/co
ntinuum.io/d/msgid/conda/CALGmxEKUe_ptQspWM4LAE077E1waqk2aBO
d37m-DmbFeAwN5HQ%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxEKUe_ptQspWM4LAE077E1waqk2aBOd37m-DmbFeAwN5HQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Erik Hovland]

But how can you handle shared libs properly while not installing deps considering the necessary shared libs are in those deps?

the issue is that conda re-writes shared lib headers

I don't know what you mean here, conda doesn't tend to rewrite any headers ever (if you mean C/C++ headers).
There may be some exceptional circumstances where a header got a prefix path baked into it I suppose but it's far
from common (and in general either harmless or more likely the right thing to do). If you mean that the dynamic
loading entries (DT_RPATH/DT_NEEDED on linux, LC_LOAD_DYLIB on macOS) in the shared libraries get
rewritten then no, that doesn't happen either since they are always stored relatively (unless they refer to system
libraries which also do not get rewritten, since prefix rewriting only affects strings that match the inserted
prefix replacement token - /opt/anaconda1/anaconda2/anaconda3 - exactly).

There are headers that are source files that get included when you
compile something. And then in every ELF based hunk there is a portion
of the file that gives important information (especially where to find
shared libraries). conda uses patchelf to rewrite that information to
point to the location of the shared library in the conda environment.
$ ldd $(which gdal_translate) | grep libspatialite
libspatialite.so.7 =>
/work/ehovland/anaconda/4.3.0-py36/envs/msp_py27/bin/../lib/libspatialite.so.7
(0x00007ff31722f000)

The previous poster was talking about how conda uses patchelf to
change the rpath compiled into the executable or shared library on
where to find the libraries it depends on.

Erik
--
Erik Hovland
***@hovland.org
http://hovland.org/

[Chris Barker - NOAA Federal]

But how can you handle shared libs properly while not installing deps considering the necessary shared libs are in those deps?

It wouldn’t work without the deps, I just don’t want them to be
installed for me by default. In a sense develop mode isn’t about
managing my environment— it’s about injecting active source into an
existing environment.

the issue is that conda re-writes shared lib headers

I don't know what you mean here, conda doesn't tend to rewrite any headers ever (if you mean C/C++ headers).

I think this got answered in another post.

-CHB

[Chris Barker - NOAA Federal]

Thanks Ray!

My point about headers was to clarify that it is a poor choice of word


Sorry about that.

on Linux you should never use ldd and on macOS you should never use otool
-L


I didn’t know that — thanks!

And yes, conda build is doing some of the work — probably the parts that
matter.

My use case is a package with a lot of extension code — and it does not
work with conda if you setup.py develop it.

I’ve hacked my around it, so honestly have no idea if conda develop would
work in my case.

Here’s my hack if anyone is curious.

https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/re_link_for_anaconda.py

-CHB


-CHB

[Chris Barker - NOAA Federal]

Is there any reason to not just make a conda-forge recipe here to build
everything from source instead of this hack?


Because I need editable mode.

Perhaps conda develop supports that now, but it didn’t when I wrote that
hack.

-CHB



On Sat, Apr 7, 2018 at 1:41 AM, Chris Barker - NOAA Federal <

Post by Chris Barker - NOAA Federal
Thanks Ray!
My point about headers was to clarify that it is a poor choice of word
Sorry about that.
on Linux you should never use ldd and on macOS you should never use otool
-L
I didn’t know that — thanks!
And yes, conda build is doing some of the work — probably the parts that
matter.
My use case is a package with a lot of extension code — and it does not
work with conda if you setup.py develop it.
I’ve hacked my around it, so honestly have no idea if conda develop would
work in my case.
Here’s my hack if anyone is curious.
https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_
gnome/re_link_for_anaconda.py
-CHB
-CHB

[Chris Withers]

Post by Chris Barker - NOAA Federal
Is there any reason to not just make a conda-forge recipe here to build
everything from source instead of this hack?

Another newbie question I'm afraid: where's the best simple recipe I can
follow to get conda-forge to build my packages?

Before that, where's the best simple recipe I can follow to develop my
conda package (picky-conda in this case) correctly?

cheers,

Chris

[Chris Barker]

Post by Chris Withers

Post by Chris Withers
Another newbie question I'm afraid: where's the best simple recipe I can
follow to get conda-forge to build my packages?

the conda-forge docs are not totally complete, but it's a good start:

https://conda-forge.org/

Post by Chris Withers
Before that, where's the best simple recipe I can follow to develop my
conda package (picky-conda in this case) correctly?

the conda-forge example recipe is a good start:

https://github.com/conda-forge/staged-recipes/tree/master/recipes/example

then there is the conda build docs, of course:

https://conda.io/docs/user-guide/tasks/build-packages/recipe.html

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Withers]

What I do want conda develop to do is handle shared libs properly —
which pip install does not do.
To set up a project environment with conda, I recommend something like
|conda create -p ./devenv --only-deps python-project-package && conda
activate ./devenv|, and then followed by |pip install -e .|.

Apologies if I'm missing something obvious, but if
python-project-package is the package you're working on, and I'm in a
git checkout such that "pip install -e ." installs it, how does "conda
create -p ./devenv --only-deps python-project-package" know to use my
current working directory as the source for python-project-package?

cheers,

Chris

[Chris Withers]

In fact, I really hate that setup.py develop and pip install -e  try to
install deps for me by default.

I find that really surprising. How would you like dependencies for a
package you're installing in "development mode" to be installed?

Having pip install auto install deps is great for the usual "Install
this third party package" use case, but develop mode is for, well,
developing, and when you are developing your code, you really should be
thinking about and managing your deps yourself.

...I do that by specifying them in setup.py and having pip install -e .
bring them in, what works best for you?

cheers,

Chris

[Chris Barker]

Post by Chris Withers

Post by Chris Barker
In fact, I really hate that setup.py develop and pip install -e try to
install deps for me by default.

I find that really surprising. How would you like dependencies for a
package you're installing in "development mode" to be installed?

usually by hand, or only-deps, or from a requirements.txt file

Having pip install auto install deps is great for the usual "Install this

Post by Chris Withers

Post by Chris Barker
third party package" use case, but develop mode is for, well, developing,
and when you are developing your code, you really should be thinking about
and managing your deps yourself.

...I do that by specifying them in setup.py and having pip install -e .
bring them in, what works best for you?

I think my distaste comes from working with conda environments, and then
pip will use pip to install deps, which I'd really rather have installed by
conda. And it's gotten better, but when I started this, a lot of times the
deps were installed by conda, but pip didn't find them, to then would
overwrite the conda version.

I'm not saying you'd never want to have pip handle your deps for you in dev
mode, but I'd rather the default be that it didn't, and you'd have to ask
for it. something like:

pip install -e --with_deps .

and setuptools is even worse -- as it will use the horrid easy_install for
deps...

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Withers]

Post by Chris Barker
In fact, I really hate that setup.py develop and pip install -e
try to install deps for me by default.
I find that really surprising. How would you like dependencies for a
package you're installing in "development mode" to be installed?
usually by hand,

That seems strangely masochistic, or indeed sadistic when it comes to
your other team members ;-)

Post by Chris Barker
or only-deps,

How does installing only-deps differ from just installing with-deps in
the first place?

Post by Chris Barker
or from a requirements.txt file

What would be in this requirements.txt? bare packages or fully locked down?

Post by Chris Barker
Having pip install auto install deps is great for the usual
"Install this third party package" use case, but develop mode is
for, well, developing, and when you are developing your code,
you really should be thinking about and managing your deps yourself.

I know I'm probably labouring the point, but install_requires is where I
manage the deps on libraries I'm developing. For apps, well, for me
that's what this whole thread has been about when it comes to conda
envs. For non-conda stuff, there's pipenv, for JS there's npm and for
rust there's cargo.

Post by Chris Barker
I think my distaste comes from working with conda environments, and then
pip will use pip to install deps, which I'd really rather have installed
by conda. And it's gotten better, but when I started this, a lot of
times the deps were installed by conda, but pip didn't find them, to
then would overwrite the conda version.

Yep, I remember this game, however these are deficiencies in conda: if
you want to support installing packages with pip, you need to do it
properly ;-)

Post by Chris Barker
I'm not saying you'd never want to have pip handle your deps for you in
dev mode, but I'd rather the default be that it didn't, and you'd have
pip install -e --with_deps .

I have a feeling you're in a minority of people who'd want the default
this way round.

Post by Chris Barker
and setuptools is even worse -- as it will use the horrid easy_install
for deps...

easy_install is dead, pretty sure setuptools doesn't install anything
with it anymore...

Chris

[Chris Barker]

We all have our different ideas and workflows -- I've said my piece. So
I'll only comment on these:

I know I'm probably labouring the point, but install_requires is where I

manage the deps on libraries I'm developing. For apps, well, for me that's
what this whole thread has been about when it comes to conda envs.

The line between a large library and an app is not always so clear....

I think my distaste comes from working with conda environments, and then

Post by Chris Barker
pip will use pip to install deps, which I'd really rather have installed by
conda. And it's gotten better, but when I started this, a lot of times the
deps were installed by conda, but pip didn't find them, to then would
overwrite the conda version.

Yep, I remember this game, however these are deficiencies in conda: if you
want to support installing packages with pip, you need to do it properly ;-)

No one ever said that a third party package manager should support pip.
Which is the whole problem -- I want to use conda (or something else) to
manage my environment, then I want to be able to do a develop install of my
python package that is under development. Because you pretty much HAVE to
use develop mode, so that your tests and example all run teh usual way
without sys.path hacking.

So I do:

setup.py develop

and get $%# easy_install trying to install my deps.

or

pip install -e

and now I get pip trying to install my deps.

What it comes down to is that setuptools is an unholy mess of build,
install, package management, and run-time features.

The PyPa is working to detangle all that, and pip is a good way toward that
goal, but you've still got a mixture of building and package management
that I'd like to see more detangled...

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Withers]

Conda develop is painful.  I highly doubt that it does anything useful
for anyone right now.  It is something I want to rewrite into something
useful, but I advise using ``python setup.py develop`` or ``pip install
https://github.com/conda/conda-build/issues/1992#issuecomment-322588270
https://github.com/conda/conda-build/issues/2633
https://github.com/conda/conda-build/issues/1677
https://github.com/conda/conda-build/issues/2791
and more...

Hmm, okay, that's a shame. Perhaps it should be removed, along with
'conda create' and 'conda update', leaving only the 'conda env {foo}'
path to do stuff with environments?

In the meantime, where can I find a good example of a development and CI
path for a conda package?

cheers,

Chris

[Chris Withers]

Post by Chris Withers
Hmm, pip freeze manages this "ok": figures out git url if there is
one or, iirc, will just use the path on disk, which can be fine for
reproducing things where that path is, say, to a shared filesystem
being used as a local repo.
I'm actually a little surprised, I would expect conda to punt to pip for
this anyway.
Sounds pragmatic and sensible, but if conda doesn't want to do the
same things that pip freeze does with "develop" or "editable"
install, shouldn't it either filter them out or warn/error?
yeah, putting something that will never work into an environment file
would be a bug. Seems worth reporting.

https://github.com/conda/conda/issues/7137

Chris