[conda] "nested" environments

['Chris Barker' via conda - Public]

I'm managing an application that is a "stack" of 4-5 related
projects/packages that have a big pile o' complex dependencies.

And I want to make conda environments for my users, so it's easy to work
with and deploy.

However, some folks may need to run the whole stack, some may need to run
only part of the stack.

So far, I've been using conda_requirements files (or an environment.yaml)
for each component.

But then how do I give someone a nice way to run the whole stack? I've
built up a single environment that runs everything, but now I have
redundancy between that "stack" environment and each individual
environment, and have to be careful to update multiple environment files
when I update a version of a dependency.

Anyone have any ideas how to make this manageable?

In theory, I could make a conda package out of each component, and then
have a meta-package that depended on all of them. But in fact, ,all the
components are under pretty active development with a fair number of
inter-dependencies -- and I want a simple systems for folks that want to
dig around in the code....

what I guess I'd like is a way to create an environment from more than one
environment file, and it would create the super-set for me.

Is that possible?

any other idea about how to manage this?

On a related note, I also have the core dependencies required to run a
given package, and then some more to test, and some more to develop

So three sets of requirements that I'd like to mix and match....

Any ideas welcome.

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

['Chris Barker' via conda - Public]

I think you need the metapackage approach,

The problem there is that I really don't want to have to have a live conda
package for the pieces of teh stack in order to run the whole thing.

Which means I still have to specify the same deps in two places -- in the
sub-pieces, and the whole stack.

Or maybe I'm missing something.

Now that I think about -- that would work with pip -- in that case, you
could do either an "editable" or regular install of the sub-pieces, and
then the meta-package would only need to see that the sub-pieces ar
installed, and it would know that the deps were all there.

Which gives me an idea -- maybe I can use "conda develop", which I hneatly
haven't tried for years, when it was not robust enough to support compiled
libs. Is conda develop robust now? Would it do what I want here? -- that
is:

conda develop sub_piece_a
conda develop sub_piece_b

would install, in develop mode, those two pieces, and all their dependencies

conda develop master_package (Or conda install)

would then see that sub_piece_a and sub_piece_b were there, and therefore
all the deps.

In short, since I may not have explained it well, I'm try to:

a) Create a set of inter-related packages
b) Be able to run them as a full stack, or individually -- all in "develop"
mode -- i.e. from source, rather than an installed conda package.
c) Be able to specify each dependency in only one place.

(b) is critical here -- if I was only trying to supply built packages for
end users, the current dependency management and a meta-pacakge would work
fine.

plus some new stuff for conda build that I'm working on. I want to create a

"conda debug" command that given a recipe will create build and host envs,
and given a package will create the test env. Normal run depends would be
the third use case, which already exists.

That does sound really useful, but I'm not sure that would quite solve my
problem.

I guess the missing piece is that I"d like to be able to install (some of)
the deps as source (develop mode). though I'm OK with doing that by hand
(or with a script) ahead of time.

I'll try to put up a PR soon.
Thanks for your thoughts.

-CHB

On Wed, Oct 24, 2018, 19:23 'Chris Barker' via conda - Public <

Post by 'Chris Barker' via conda - Public
I'm managing an application that is a "stack" of 4-5 related
projects/packages that have a big pile o' complex dependencies.
And I want to make conda environments for my users, so it's easy to work
with and deploy.
However, some folks may need to run the whole stack, some may need to run
only part of the stack.
So far, I've been using conda_requirements files (or an environment.yaml)
for each component.
But then how do I give someone a nice way to run the whole stack? I've
built up a single environment that runs everything, but now I have
redundancy between that "stack" environment and each individual
environment, and have to be careful to update multiple environment files
when I update a version of a dependency.
Anyone have any ideas how to make this manageable?
In theory, I could make a conda package out of each component, and then
have a meta-package that depended on all of them. But in fact, ,all the
components are under pretty active development with a fair number of
inter-dependencies -- and I want a simple systems for folks that want to
dig around in the code....
what I guess I'd like is a way to create an environment from more than
one environment file, and it would create the super-set for me.
Is that possible?
any other idea about how to manage this?
On a related note, I also have the core dependencies required to run a
given package, and then some more to test, and some more to develop
So three sets of requirements that I'd like to mix and match....
Any ideas welcome.
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE (206
<https://maps.google.com/?q=7600+Sand+Point+Way+NE+%C2%A0%C2%A0(206&entry=gmail&source=g>)
526-6329 fax
Seattle, WA 98115 [phone] main reception
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CALGmxE%2BEtm9NnYagBno2xop_
VvBZMvVnP0nHbyuesHoC9V2Esg%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxE%2BEtm9NnYagBno2xop_VvBZMvVnP0nHbyuesHoC9V2Esg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

['Chris Barker' via conda - Public]

Conda develop has not seen any attention in a very long time.

I was afraid of that.

If you'd like to work on that, PRs would be most welcome.

Hmm -- maybe -- I am kinda sick of the kludge I'm using now to get
"setup.py develop" to work with compiled extensions on the Mac...

I believe the conda+pip interop stuff that Kale has been working on may
help this situation a lot. That's in the conda 4.6.0 beta, which is
available on the conda-canary channel.

OK -- I'll look at that at some point.

Having development installs of deps is only going to work when conda
recognizes that development install, can get its version accurately, and
factors that into any solution.

yup.

I agree that "conda debug" is not really a solution here - maybe in tandem
with the conda+pip interop, it would be useful.

hmm -- maybe (and given that my stack still a few deps that are not yet
conda-available -- that could help too)

I think maybe I'll need to have some hacky scripts that build the
dependency files from a single source -- not a general solution but maybe
helpful.

Thinking out loud here -- conda has packages that depend on other packages
that depend on ...

Maybe there could be the concept of an environment dependency. So in my
case:

I'd create an environment file(s) for the base package(s) that could be
used by themselves.

Then I'd create a super-environment spec that would depend one or two
environment files, and add a few more packages in there.

maybe it would be as "simple" as allowing:

conda env create -f environ1.yml environ2.yml environ3.yml

Which I will confess I've never tried...

-CHB

Post by 'Chris Barker' via conda - Public

I think you need the metapackage approach,

The problem there is that I really don't want to have to have a live
conda package for the pieces of teh stack in order to run the whole thing.
Which means I still have to specify the same deps in two places -- in the
sub-pieces, and the whole stack.
Or maybe I'm missing something.
Now that I think about -- that would work with pip -- in that case, you
could do either an "editable" or regular install of the sub-pieces, and
then the meta-package would only need to see that the sub-pieces ar
installed, and it would know that the deps were all there.
Which gives me an idea -- maybe I can use "conda develop", which I
hneatly haven't tried for years, when it was not robust enough to support
compiled libs. Is conda develop robust now? Would it do what I want here?
conda develop sub_piece_a
conda develop sub_piece_b
would install, in develop mode, those two pieces, and all their dependencies
conda develop master_package (Or conda install)
would then see that sub_piece_a and sub_piece_b were there, and therefore
all the deps.
a) Create a set of inter-related packages
b) Be able to run them as a full stack, or individually -- all in
"develop" mode -- i.e. from source, rather than an installed conda package.
c) Be able to specify each dependency in only one place.
(b) is critical here -- if I was only trying to supply built packages for
end users, the current dependency management and a meta-pacakge would work
fine.
plus some new stuff for conda build that I'm working on. I want to create

a "conda debug" command that given a recipe will create build and host
envs, and given a package will create the test env. Normal run depends
would be the third use case, which already exists.

That does sound really useful, but I'm not sure that would quite solve my
problem.
I guess the missing piece is that I"d like to be able to install (some
of) the deps as source (develop mode). though I'm OK with doing that by
hand (or with a script) ahead of time.
I'll try to put up a PR soon.
Thanks for your thoughts.
-CHB

On Wed, Oct 24, 2018, 19:23 'Chris Barker' via conda - Public <

Post by 'Chris Barker' via conda - Public
I'm managing an application that is a "stack" of 4-5 related
projects/packages that have a big pile o' complex dependencies.
And I want to make conda environments for my users, so it's easy to
work with and deploy.
However, some folks may need to run the whole stack, some may need to
run only part of the stack.
So far, I've been using conda_requirements files (or an
environment.yaml) for each component.
But then how do I give someone a nice way to run the whole stack? I've
built up a single environment that runs everything, but now I have
redundancy between that "stack" environment and each individual
environment, and have to be careful to update multiple environment files
when I update a version of a dependency.
Anyone have any ideas how to make this manageable?
In theory, I could make a conda package out of each component, and then
have a meta-package that depended on all of them. But in fact, ,all the
components are under pretty active development with a fair number of
inter-dependencies -- and I want a simple systems for folks that want to
dig around in the code....
what I guess I'd like is a way to create an environment from more than
one environment file, and it would create the super-set for me.
Is that possible?
any other idea about how to manage this?
On a related note, I also have the core dependencies required to run a
given package, and then some more to test, and some more to develop
So three sets of requirements that I'd like to mix and match....
Any ideas welcome.
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE (206
<https://maps.google.com/?q=7600+Sand+Point+Way+NE+%C2%A0%C2%A0(206&entry=gmail&source=g>)
526-6329 fax
Seattle, WA 98115 [phone] main reception
--
You received this message because you are subscribed to the Google
Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/a/
continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CALGmxE%2BEtm9NnYagBno2xop_
VvBZMvVnP0nHbyuesHoC9V2Esg%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxE%2BEtm9NnYagBno2xop_VvBZMvVnP0nHbyuesHoC9V2Esg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/
continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE (206
<https://maps.google.com/?q=7600+Sand+Point+Way+NE+%C2%A0%C2%A0(206&entry=gmail&source=g>)
526-6329 fax
Seattle, WA 98115 [phone] main reception

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Bryce DeAlessio]

Hello I am curious if possibly anaconda-project https://github.com/Anaconda-Platform/anaconda-project may actually fit your needs here. Its billed as a way to consistently setup data science projects but it seems it may have more general purpose use cases as well as an automated way to set up conda environments.

['Chris Barker' via conda - Public]

Post by Bryce DeAlessio
Hello I am curious if possibly anaconda-project
https://github.com/Anaconda-Platform/anaconda-project may actually fit
your needs here. Its billed as a way to consistently setup data science
projects but it seems it may have more general purpose use cases as well as
an automated way to set up conda environments.

thanks for the tip.

No, it doesn't appear solve the problem I was asking about -- it's still
about one big project, whereas I have one big project, and sub-projects
that I want folks to be able to use independently.

On the other hand, is does look like it could be a nice way to setup and
run the one big project :-)

It also doesn't look like it's been touched in a year or so -- anyone know
the status?

-CHB

Post by Bryce DeAlessio
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/24979af0-698e-4c40-8d2a-
f33b792970ae%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Nehal J Wani]

Post by 'Chris Barker' via conda - Public
conda env create -f environ1.yml environ2.yml environ3.yml

Instead of this, try:
conda create -n env --file environ1.txt --file environ2.txt --file
environ3.txt

...and make sure that intersection of the contents in these files is null.

ProjA guys can use environ1.txt
ProjB guys can use environ1.txt and environ2.txt
ProjC guys who want the whole stack, can use all three.

Cheers!

Nehal J Wani


On Fri, Nov 2, 2018 at 3:54 AM 'Chris Barker' via conda - Public <

Post by 'Chris Barker' via conda - Public

Post by Bryce DeAlessio
Hello I am curious if possibly anaconda-project
https://github.com/Anaconda-Platform/anaconda-project may actually fit
your needs here. Its billed as a way to consistently setup data science
projects but it seems it may have more general purpose use cases as well as
an automated way to set up conda environments.

thanks for the tip.
No, it doesn't appear solve the problem I was asking about -- it's still
about one big project, whereas I have one big project, and sub-projects
that I want folks to be able to use independently.
On the other hand, is does look like it could be a nice way to setup and
run the one big project :-)
It also doesn't look like it's been touched in a year or so -- anyone know
the status?
-CHB

Post by Bryce DeAlessio
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/24979af0-698e-4c40-8d2a-f33b792970ae%40continuum.io
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxE%2BxP3hPB73FmG%3DpCe20d5GpZzYxdjA69eBdjkG7SK5W2A%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxE%2BxP3hPB73FmG%3DpCe20d5GpZzYxdjA69eBdjkG7SK5W2A%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

['Chris Barker' via conda - Public]

Post by Nehal J Wani

Post by 'Chris Barker' via conda - Public
conda env create -f environ1.yml environ2.yml environ3.yml

conda create -n env --file environ1.txt --file environ2.txt --file
environ3.txt

cool -- I'll give that a shot.

Post by Nehal J Wani
...and make sure that intersection of the contents in these files is null.

if it works, then maybe I'll work on a tool to create the non-intersecting
files. That'll be a big awkward to create by hand.

Thanks,

-CHB

Post by Nehal J Wani
ProjA guys can use environ1.txt
ProjB guys can use environ1.txt and environ2.txt
ProjC guys who want the whole stack, can use all three.
Cheers!
Nehal J Wani
On Fri, Nov 2, 2018 at 3:54 AM 'Chris Barker' via conda - Public <

Post by 'Chris Barker' via conda - Public

Post by Bryce DeAlessio
Hello I am curious if possibly anaconda-project
https://github.com/Anaconda-Platform/anaconda-project may actually fit
your needs here. Its billed as a way to consistently setup data science
projects but it seems it may have more general purpose use cases as well as
an automated way to set up conda environments.

thanks for the tip.
No, it doesn't appear solve the problem I was asking about -- it's still
about one big project, whereas I have one big project, and sub-projects
that I want folks to be able to use independently.
On the other hand, is does look like it could be a nice way to setup and
run the one big project :-)
It also doesn't look like it's been touched in a year or so -- anyone
know the status?
-CHB

Post by Bryce DeAlessio
--
You received this message because you are subscribed to the Google
Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/a/
continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/24979af0-698e-4c40-8d2a-
f33b792970ae%40continuum.io.
For more options, visit https://groups.google.com/a/
continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE (206
<https://maps.google.com/?q=7600+Sand+Point+Way+NE+%C2%A0%C2%A0(206&entry=gmail&source=g>)
526-6329 fax
Seattle, WA 98115 [phone] main reception
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/
.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CALGmxE%2BxP3hPB73FmG%
3DpCe20d5GpZzYxdjA69eBdjkG7SK5W2A%40mail.gmail.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CALGmxE%2BxP3hPB73FmG%3DpCe20d5GpZzYxdjA69eBdjkG7SK5W2A%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout
.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov