[conda] export environment to an offline machine

[Jonathan MacCarthy]

Hello all,

I'm trying to build a single tarball on a machine with internet such that I
can reproduce my "environment" from scratch on a second offline machine
(both linux-64). I feel like I'm doing it wrong, and I'm looking for
help. Here's my current proceedure.

On the online (source) machine:
1. Install miniconda and conda-build
2. "conda install thing1 thing2 ..." all the things I can from conda's
servers (this caches a bunch of tar.bz2 files that I can use on the second
machine)
3. For the rest of my required packages, download their zip files so that I
can "pip install" them on the other machine
4. Make a poor-man's local conda channel: mkdir -p my_channel/linux-64 &&
cp anaconda2/pkgs/*.bz2 my_channel/linux-64/
5. Index the channel: conda index my_channel/linux-64
6. tar/zip the channel directory, miniconda, and the pip zip files, and
send it to the offline machine

On the offline (destination) machine:
1. unzip/untar the archive
2. Install miniconda
3. "conda install -c file://full/path/to/my_channel --override-channels
thing1 thing2 ..." in the same order as before
4. "pip install --no-deps" the other zip files
5. profit


This seems like a LOT of steps. Is there a better way to do this? I
haven't found it yet.

I think both "conda list -e" and "conda env export" assume that the
destination machine has internet. Also, this doesn't work very well if I'm
starting from inside an environment on the source/online machine, because
all of miniconda's tar.bz2 files for all environments go into
miniconda2/pkgs, and there seems to be no built-in way to grep out only the
ones I just installed in my environment.

Suggestions are most welcomed.

Best,
Jon

[Travis Oliphant]

1) You don't have to make a conda channel
2) You can create a conda package out of all the stuff in your current
environment (i.e. that was installed without conda)

conda package --pkg-name NAME --pkg-version VERSION --pkg-build BUILD

All the *untracked* files in your environment will end up in a conda
package with the NAME-VERSION-BUILD you specify.

run

conda package -u

to see what will be in that file.


3) You can build a single large .tar file that contains all the
*conda-packages* you want to include

Just copy all the conda packages into a directory and run .tar

4) conda install <somefilename>.tar

will grab all the conda packages in the .tar file and install them.

You can hand the .tar file around and easily install it anywhere.

-Travis

Post by Jonathan MacCarthy
Hello all,
I'm trying to build a single tarball on a machine with internet such that
I can reproduce my "environment" from scratch on a second offline machine
(both linux-64). I feel like I'm doing it wrong, and I'm looking for
help. Here's my current proceedure.
1. Install miniconda and conda-build
2. "conda install thing1 thing2 ..." all the things I can from conda's
servers (this caches a bunch of tar.bz2 files that I can use on the second
machine)
3. For the rest of my required packages, download their zip files so that
I can "pip install" them on the other machine
4. Make a poor-man's local conda channel: mkdir -p my_channel/linux-64 &&
cp anaconda2/pkgs/*.bz2 my_channel/linux-64/
5. Index the channel: conda index my_channel/linux-64
6. tar/zip the channel directory, miniconda, and the pip zip files, and
send it to the offline machine
1. unzip/untar the archive
2. Install miniconda
3. "conda install -c file://full/path/to/my_channel --override-channels
thing1 thing2 ..." in the same order as before
4. "pip install --no-deps" the other zip files
5. profit
This seems like a LOT of steps. Is there a better way to do this? I
haven't found it yet.
I think both "conda list -e" and "conda env export" assume that the
destination machine has internet. Also, this doesn't work very well if I'm
starting from inside an environment on the source/online machine, because
all of miniconda's tar.bz2 files for all environments go into
miniconda2/pkgs, and there seems to be no built-in way to grep out only the
ones I just installed in my environment.
Suggestions are most welcomed.
Best,
Jon
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.

--
*Travis Oliphant, PhD*
*Co-founder and CEO*


@teoliphant
512-222-5440
http://www.continuum.io

[PeterC]

Please bear with me as I am new to packaging.

I have a python/cython/c application with a list of both conda and pip
installed dependencies. I've got a conda recipe and can create a conda
package that installs on a dev machine from --use-local or from a custom
channel.

I'd like to be able to distribute a single tar file that can be conda
installed on user's machine. However, it can't be conda installed from the
package I build because conda install doesn't check dependencies when it
installs from a .bz2 file. It appears I need to wrap it in a channel to
trigger conda install to check dependencies.

As a result, I have been going down the path of creating a custom channel
in a dir, zipping that dir up, sending the zipped dir with scripts to unzip
and conda install from that dir (channel). To handle the pip installs I
have been either creating conda packages of them or using a pre-link.sh or
pre-link.bat file to install them.

The plan is to setup a three different build envs to support windows, osx
and linux because of the compiler bits for cython/c. However, for now I'm
just working on try to get a win-64 build and distribution up in conda.

I think that wrapping my .bz2 file in a channel will work fine but if I
could get a complete tar of a working env and dist that I think I'd be even
happier.

This may not be possible but I thought it was worth asking. Is it possible
to create from a working conda env on your build machine a single tar of
that working env so that it contains all the actual files (not just
dependency info) so that it can be distributed as a single file and
installed into a user's env using conda install from the tar file?

Travis' approach looks close to this but handles the pip and conda bits
separately. If you have to manually create the conda and pip installed
parts done separately, where exactly do you get the actual files to tar up
from? Do you just copy the appropriate sub-dirs from the dev env's
site-packages dir?

Many thanks, Peter

[Tristan]

Hi PeterC,

I am currently facing the same issue and am trying to achieve quite exactly
the same.
Did you find a good solution?

Regards and thanks,
Tristan

Post by PeterC
Please bear with me as I am new to packaging.
I have a python/cython/c application with a list of both conda and pip
installed dependencies. I've got a conda recipe and can create a conda
package that installs on a dev machine from --use-local or from a custom
channel.
I'd like to be able to distribute a single tar file that can be conda
installed on user's machine. However, it can't be conda installed from the
package I build because conda install doesn't check dependencies when it
installs from a .bz2 file. It appears I need to wrap it in a channel to
trigger conda install to check dependencies.
As a result, I have been going down the path of creating a custom channel
in a dir, zipping that dir up, sending the zipped dir with scripts to unzip
and conda install from that dir (channel). To handle the pip installs I
have been either creating conda packages of them or using a pre-link.sh or
pre-link.bat file to install them.
The plan is to setup a three different build envs to support windows, osx
and linux because of the compiler bits for cython/c. However, for now I'm
just working on try to get a win-64 build and distribution up in conda.
I think that wrapping my .bz2 file in a channel will work fine but if I
could get a complete tar of a working env and dist that I think I'd be even
happier.
This may not be possible but I thought it was worth asking. Is it possible
to create from a working conda env on your build machine a single tar of
that working env so that it contains all the actual files (not just
dependency info) so that it can be distributed as a single file and
installed into a user's env using conda install from the tar file?
Travis' approach looks close to this but handles the pip and conda bits
separately. If you have to manually create the conda and pip installed
parts done separately, where exactly do you get the actual files to tar up
from? Do you just copy the appropriate sub-dirs from the dev env's
site-packages dir?
Many thanks, Peter

[Lori Burns]

Have you considered a conda constructor
(https://github.com/conda/constructor)? Works like the miniconda installers
but also includes whatever packages you specify.

Post by Tristan
Hi PeterC,
I am currently facing the same issue and am trying to achieve quite
exactly the same.
Did you find a good solution?
Regards and thanks,
Tristan

Post by PeterC
Please bear with me as I am new to packaging.
I have a python/cython/c application with a list of both conda and pip
installed dependencies. I've got a conda recipe and can create a conda
package that installs on a dev machine from --use-local or from a custom
channel.
I'd like to be able to distribute a single tar file that can be conda
installed on user's machine. However, it can't be conda installed from the
package I build because conda install doesn't check dependencies when it
installs from a .bz2 file. It appears I need to wrap it in a channel to
trigger conda install to check dependencies.
As a result, I have been going down the path of creating a custom channel
in a dir, zipping that dir up, sending the zipped dir with scripts to unzip
and conda install from that dir (channel). To handle the pip installs I
have been either creating conda packages of them or using a pre-link.sh or
pre-link.bat file to install them.
The plan is to setup a three different build envs to support windows, osx
and linux because of the compiler bits for cython/c. However, for now I'm
just working on try to get a win-64 build and distribution up in conda.
I think that wrapping my .bz2 file in a channel will work fine but if I
could get a complete tar of a working env and dist that I think I'd be even
happier.
This may not be possible but I thought it was worth asking. Is it
possible to create from a working conda env on your build machine a single
tar of that working env so that it contains all the actual files (not just
dependency info) so that it can be distributed as a single file and
installed into a user's env using conda install from the tar file?
Travis' approach looks close to this but handles the pip and conda bits
separately. If you have to manually create the conda and pip installed
parts done separately, where exactly do you get the actual files to tar up
from? Do you just copy the appropriate sub-dirs from the dev env's
site-packages dir?
Many thanks, Peter

[Tristan]

I actually have looked into the conda constructor way - but I felt it
provided too much unneeded functionality:
Is there a way to use the constructor to just add an environment with a
specified list of packages to the existing conda installation on the local
machine?
The only way it worked for me, was that it kind of created a whole new,
seperate Conda installation.
Maybe you can help me on that? Thanks!

Have you considered a conda constructor (
https://github.com/conda/constructor)? Works like the miniconda
installers but also includes whatever packages you specify.

Post by Tristan
Hi PeterC,
I am currently facing the same issue and am trying to achieve quite
exactly the same.
Did you find a good solution?
Regards and thanks,
Tristan

Post by PeterC
Please bear with me as I am new to packaging.
I have a python/cython/c application with a list of both conda and pip
installed dependencies. I've got a conda recipe and can create a conda
package that installs on a dev machine from --use-local or from a custom
channel.
I'd like to be able to distribute a single tar file that can be conda
installed on user's machine. However, it can't be conda installed from the
package I build because conda install doesn't check dependencies when it
installs from a .bz2 file. It appears I need to wrap it in a channel to
trigger conda install to check dependencies.
As a result, I have been going down the path of creating a custom
channel in a dir, zipping that dir up, sending the zipped dir with scripts
to unzip and conda install from that dir (channel). To handle the pip
installs I have been either creating conda packages of them or using a
pre-link.sh or pre-link.bat file to install them.
The plan is to setup a three different build envs to support windows,
osx and linux because of the compiler bits for cython/c. However, for now
I'm just working on try to get a win-64 build and distribution up in conda.
I think that wrapping my .bz2 file in a channel will work fine but if I
could get a complete tar of a working env and dist that I think I'd be even
happier.
This may not be possible but I thought it was worth asking. Is it
possible to create from a working conda env on your build machine a single
tar of that working env so that it contains all the actual files (not just
dependency info) so that it can be distributed as a single file and
installed into a user's env using conda install from the tar file?
Travis' approach looks close to this but handles the pip and conda bits
separately. If you have to manually create the conda and pip installed
parts done separately, where exactly do you get the actual files to tar up
from? Do you just copy the appropriate sub-dirs from the dev env's
site-packages dir?
Many thanks, Peter

[Jonathan MacCarthy]

+1 on this; I'd just like to create and fill an environment without the internet. I need a way to export an existing environment, including the bz2 files.

-Jon

________________________________
From: Tristan <***@gmail.com>
Sent: Wednesday, August 30, 2017 8:52:01 AM
To: conda - Public
Cc: ***@gmail.com; ***@continuum.io
Subject: Re: [conda] export environment to an offline machine

I actually have looked into the conda constructor way - but I felt it provided too much unneeded functionality:
Is there a way to use the constructor to just add an environment with a specified list of packages to the existing conda installation on the local machine?
The only way it worked for me, was that it kind of created a whole new, seperate Conda installation.
Maybe you can help me on that? Thanks!

Am Dienstag, 29. August 2017 19:11:58 UTC+2 schrieb Lori Burns:
Have you considered a conda constructor (https://github.com/conda/constructor)? Works like the miniconda installers but also includes whatever packages you specify.

On Tuesday, August 29, 2017 at 7:43:26 AM UTC-4, Tristan wrote:
Hi PeterC,

I am currently facing the same issue and am trying to achieve quite exactly the same.
Did you find a good solution?

Regards and thanks,
Tristan

Am Freitag, 1. April 2016 04:38:40 UTC+2 schrieb PeterC:
Please bear with me as I am new to packaging.

I have a python/cython/c application with a list of both conda and pip installed dependencies. I've got a conda recipe and can create a conda package that installs on a dev machine from --use-local or from a custom channel.

I'd like to be able to distribute a single tar file that can be conda installed on user's machine. However, it can't be conda installed from the package I build because conda install doesn't check dependencies when it installs from a .bz2 file. It appears I need to wrap it in a channel to trigger conda install to check dependencies.

As a result, I have been going down the path of creating a custom channel in a dir, zipping that dir up, sending the zipped dir with scripts to unzip and conda install from that dir (channel). To handle the pip installs I have been either creating conda packages of them or using a pre-link.sh or pre-link.bat file to install them.

The plan is to setup a three different build envs to support windows, osx and linux because of the compiler bits for cython/c. However, for now I'm just working on try to get a win-64 build and distribution up in conda.

I think that wrapping my .bz2 file in a channel will work fine but if I could get a complete tar of a working env and dist that I think I'd be even happier.

This may not be possible but I thought it was worth asking. Is it possible to create from a working conda env on your build machine a single tar of that working env so that it contains all the actual files (not just dependency info) so that it can be distributed as a single file and installed into a user's env using conda install from the tar file?

Travis' approach looks close to this but handles the pip and conda bits separately. If you have to manually create the conda and pip installed parts done separately, where exactly do you get the actual files to tar up from? Do you just copy the appropriate sub-dirs from the dev env's site-packages dir?

Many thanks, Peter

[Chris Barker]

Post by Jonathan MacCarthy
+1 on this; I'd just like to create and fill an environment without the
internet. I need a way to export an existing environment, including the
bz2 files.

kindof by definition, a environment doesn't have the bz2 files -- it has
the installed packages.

But:

as far as I can tell, when you install a package into an environment, it
gets put in the central collection of pacakges. IN my miniconda install,
this is:

~/miniconda2/pkgs

That dir has all the bz2 files and unpacked directories for (I think) every
package I've installed since I installed miniconda (I think there are way
sot purge it though, if it gets too big with kruft...)

So if you:

install a new miniconda

install all the packages you want and/or create an environment from a
environment.yaml file

you can then zip up the miniconda/pkgs dir, and you'll have everything you
need -- you can probably dump it by hand into the pkgs dir on another
install, and be set to go.

Or be a bit smarter, and do a:

conda list

in an environment, and then go through that list and grab the packages from
the /pkgs dir, and pack those up.

unpack them into the other pkgs dir, and you're set to go.

-CHB

Post by Jonathan MacCarthy
-Jon
------------------------------
*Sent:* Wednesday, August 30, 2017 8:52:01 AM
*To:* conda - Public
*Subject:* Re: [conda] export environment to an offline machine
I actually have looked into the conda constructor way - but I felt it
Is there a way to use the constructor to just add an environment with a
specified list of packages to the existing conda installation on the local
machine?
The only way it worked for me, was that it kind of created a whole new,
seperate Conda installation.
Maybe you can help me on that? Thanks!

Post by Jonathan MacCarthy
Have you considered a conda constructor (https://github.com/conda/cons
tructor)? Works like the miniconda installers but also includes whatever
packages you specify.

Post by Tristan
Hi PeterC,
I am currently facing the same issue and am trying to achieve quite exactly the same.
Did you find a good solution?
Regards and thanks,
Tristan

Post by PeterC
Please bear with me as I am new to packaging.
I have a python/cython/c application with a list of both conda and pip
installed dependencies. I've got a conda recipe and can create a conda
package that installs on a dev machine from --use-local or from a custom
channel.
I'd like to be able to distribute a single tar file that can be conda
installed on user's machine. However, it can't be conda installed from the
package I build because conda install doesn't check dependencies when it
installs from a .bz2 file. It appears I need to wrap it in a channel to
trigger conda install to check dependencies.
As a result, I have been going down the path of creating a custom
channel in a dir, zipping that dir up, sending the zipped dir with scripts
to unzip and conda install from that dir (channel). To handle the pip
installs I have been either creating conda packages of them or using a
pre-link.sh or pre-link.bat file to install them.
The plan is to setup a three different build envs to support windows,
osx and linux because of the compiler bits for cython/c. However, for now
I'm just working on try to get a win-64 build and distribution up in conda.
I think that wrapping my .bz2 file in a channel will work fine but if I
could get a complete tar of a working env and dist that I think I'd be even
happier.
This may not be possible but I thought it was worth asking. Is it
possible to create from a working conda env on your build machine a single
tar of that working env so that it contains all the actual files (not just
dependency info) so that it can be distributed as a single file and
installed into a user's env using conda install from the tar file?
Travis' approach looks close to this but handles the pip and conda bits
separately. If you have to manually create the conda and pip installed
parts done separately, where exactly do you get the actual files to tar up
from? Do you just copy the appropriate sub-dirs from the dev env's
site-packages dir?
Many thanks, Peter
--

You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CY4PR22MB0968FA29C71359AF6A62B
D1EAC9C0%40CY4PR22MB0968.namprd22.prod.outlook.com
<https://groups.google.com/a/continuum.io/d/msgid/conda/CY4PR22MB0968FA29C71359AF6A62BD1EAC9C0%40CY4PR22MB0968.namprd22.prod.outlook.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Rob G]

Post by Travis Oliphant
3) You can build a single large .tar file that contains all the
*conda-packages* you want to include
Just copy all the conda packages into a directory and run .tar
4) conda install <somefilename>.tar
will grab all the conda packages in the .tar file and install them.
You can hand the .tar file around and easily install it anywhere.
*Travis Oliphant, PhD*
*Co-founder and CEO*
@teoliphant
512-222-5440
http://www.continuum.io

When you say "conda packages", you're not referring to the .tar.bz2 files
are you? Because I tried that and it didn't work.

[Zhen Yuan Yeo]

Not sure if anyone still has a use for this, but I have recently tried to
install packages on an offline computer. these are the steps which worked
for me, even though it may seem "hacky".
On the computer which you want to copy the packages *from* (I'm using
miniconda3):
1. go to the installation folder and look for the folder pkgs. for me its
in /home/username/miniconda3/pkgs
2. copy the desired packages onto a temporary folder, ie
imageio-2.2.0-py36he555465_0.tar.bz2 or ( *.tar.bz2 ). you dont need to
copy the folders, ie, imageio-2.2.0-py36he555465_0.tar.bz2

2a. you can copy *all* pkgs by running this script, while in the pkgs
folder. (make sure you mkdir temp_folder)

for f in `ls *.bz2`

do

cp $f temp_folder/$f

done

3. move the temporary folder onto the offline computer (or computer without
internet access), (via scp or thumbdrive)

On the computer where you want *to install the packages*:
1. do a conda install for each package in the temporary folder (cd into the
folder and run this shell script)

for f in `ls *.bz2`

do

conda install -y $f

done


note that in step 2 you could have copied the whole pkgs folder, but it may
take a longer time as the folders are quite deep and irrelevant for the
offline installation.

On Saturday, February 13, 2016 at 6:15:40 AM UTC+8, Jonathan MacCarthy

Post by Jonathan MacCarthy
Hello all,
I'm trying to build a single tarball on a machine with internet such that
I can reproduce my "environment" from scratch on a second offline machine
(both linux-64). I feel like I'm doing it wrong, and I'm looking for
help. Here's my current proceedure.
1. Install miniconda and conda-build
2. "conda install thing1 thing2 ..." all the things I can from conda's
servers (this caches a bunch of tar.bz2 files that I can use on the second
machine)
3. For the rest of my required packages, download their zip files so that
I can "pip install" them on the other machine
4. Make a poor-man's local conda channel: mkdir -p my_channel/linux-64 &&
cp anaconda2/pkgs/*.bz2 my_channel/linux-64/
5. Index the channel: conda index my_channel/linux-64
6. tar/zip the channel directory, miniconda, and the pip zip files, and
send it to the offline machine
1. unzip/untar the archive
2. Install miniconda
3. "conda install -c file://full/path/to/my_channel --override-channels
thing1 thing2 ..." in the same order as before
4. "pip install --no-deps" the other zip files
5. profit
This seems like a LOT of steps. Is there a better way to do this? I
haven't found it yet.
I think both "conda list -e" and "conda env export" assume that the
destination machine has internet. Also, this doesn't work very well if I'm
starting from inside an environment on the source/online machine, because
all of miniconda's tar.bz2 files for all environments go into
miniconda2/pkgs, and there seems to be no built-in way to grep out only the
ones I just installed in my environment.
Suggestions are most welcomed.
Best,
Jon

[Zhen Yuan Yeo]

Not sure if anyone still has a use for this, but I have recently tried to
install packages on an offline computer. these are the steps which worked
for me, even though it may seem "hacky".
On the computer which you want to copy the packages *from* (I'm using
miniconda3):
1. go to the installation folder and look for the folder pkgs. for me its
in /home/username/miniconda3/pkgs
2. copy the desired packages onto a temporary folder, ie
imageio-2.2.0-py36he555465_0.tar.bz2 or ( *.tar.bz2 ). you dont need to
copy the folders, ie, imageio-2.2.0-py36he555465_0/

2a. you can copy *all* pkgs by running this script, while in the pkgs
folder. (make sure you mkdir temp_folder)

for f in `ls *.bz2`

do

cp $f temp_folder/$f

done

3. move the temporary folder onto the offline computer (or computer without
internet access), (via scp or thumbdrive)

On the computer where you want *to install the packages*:
1. do a conda install for each package in the temporary folder (cd into the
folder and run this shell script)

for f in `ls *.bz2`

do

conda install -y $f

done


note that in step 2 you could have copied the whole pkgs folder, but it may
take a longer time as the folders are quite deep and irrelevant for the
offline installation.

On Saturday, February 13, 2016 at 6:15:40 AM UTC+8, Jonathan MacCarthy

Post by Jonathan MacCarthy
Hello all,
I'm trying to build a single tarball on a machine with internet such that
I can reproduce my "environment" from scratch on a second offline machine
(both linux-64). I feel like I'm doing it wrong, and I'm looking for
help. Here's my current proceedure.
1. Install miniconda and conda-build
2. "conda install thing1 thing2 ..." all the things I can from conda's
servers (this caches a bunch of tar.bz2 files that I can use on the second
machine)
3. For the rest of my required packages, download their zip files so that
I can "pip install" them on the other machine
4. Make a poor-man's local conda channel: mkdir -p my_channel/linux-64 &&
cp anaconda2/pkgs/*.bz2 my_channel/linux-64/
5. Index the channel: conda index my_channel/linux-64
6. tar/zip the channel directory, miniconda, and the pip zip files, and
send it to the offline machine
1. unzip/untar the archive
2. Install miniconda
3. "conda install -c file://full/path/to/my_channel --override-channels
thing1 thing2 ..." in the same order as before
4. "pip install --no-deps" the other zip files
5. profit
This seems like a LOT of steps. Is there a better way to do this? I
haven't found it yet.
I think both "conda list -e" and "conda env export" assume that the
destination machine has internet. Also, this doesn't work very well if I'm
starting from inside an environment on the source/online machine, because
all of miniconda's tar.bz2 files for all environments go into
miniconda2/pkgs, and there seems to be no built-in way to grep out only the
ones I just installed in my environment.
Suggestions are most welcomed.
Best,
Jon

[D3VIATION]

Have you looked at using Docker with Conda. We use it to share environments and some what lock it as well for reproducibility.