[conda] Confused about data file locations

[Skip Montanaro]

With a bit of reading I think I determined how to specify data files
to include in setup.py:

setuptools.setup(

...
data_files=[('relative/dir', ['data/mumble.ini'])],
...
)

When I build my conda package, I find mumble.ini here (using python2.7
as an example):

lib/python2.7/site-packages/relative/dir/mumble.ini
bin/run-my-service.sh

I think that's an odd place to put a data file, especially one which
(in my case) names a config file used by my start script installed in
"bin" (at the same level as "lib"). Is that common practice? Is there
a better way to do this?

Thanks,

Skip Montanaro

[Skip Montanaro]

Thanks. So, the recipe would be to execute setup.py from build.sh in
the usual fashion, then copy data files from where they reside to
${PREFIX}/some/where after that? Apologies for being a dunce in this
regard. I thought most/all package building was actually done by
setuptools, and that the Conda bits existed outside of that to sugar
coat things.

Anybody here have a concrete example they can point me to?

Skip

This is one of many reasons I hate dealing with data files in setup.py.
If you have no reason to support anything but conda packages, I recommend
just doing simple copying/moving to $PREFIX in your build.sh script.
Otherwise, may your google searches be productive - I have only ever found
dealing with data files outside of site-packages to be extraordinarily
painful. It's a Python problem, not a conda problem.

Post by Skip Montanaro
With a bit of reading I think I determined how to specify data files
setuptools.setup(
...
data_files=[('relative/dir', ['data/mumble.ini'])],
...
)
When I build my conda package, I find mumble.ini here (using python2.7
lib/python2.7/site-packages/relative/dir/mumble.ini
bin/run-my-service.sh
I think that's an odd place to put a data file, especially one which
(in my case) names a config file used by my start script installed in
"bin" (at the same level as "lib"). Is that common practice? Is there
a better way to do this?
Thanks,
Skip Montanaro
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit
https://groups.google.com/a/continuum.io/d/msgid/conda/CANc-5Uzov7y9Opt4u65zfu%2B-gqHxE-wpC97toJ1gAOW%3DnTK8GQ%40mail.gmail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Chris Barker]

If you have a pure-python package, and you want to avoid setup.py
altogether, nothing stopping you there.

no -- but the whole setup.py / setuptools / pip mess is the standard way to
do python packaging -- it'll be a lot easier if you use those tools :-)

My philosophy is to try (it's hard!) to use ONLY the build features of
setuptools -- that will make it easier.

setuptools is an unfortunate mixture of build-time, install-time, and
run-time features. PyPa has been trying to untangle that for years, and
they are getting close, but setuptools itself still contains all those
features, so it's a bit hard to avoid hitting them accidentally :-(

Anyway, there are standard ways to install data files with python packages,
you might as well use one of them.

Most people suffer through figuring out the packaging with setup.py,
because they want to distribute wheels, too.

no -- because they want their packages to be compatible with the rest of
the python packing ecosystem -- i.e. satisfy dependencies, etc. And, yes to
be installed outside of conda, but not necessarily with wheels -- source
installs are a really important use case. And development.

With the meta-data peps now in place (or almost in place), maybe conda will
be abel to directly support a proper install of a python package in a
compatible way, but it's not easy now.



-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Barker]

Sure, good point. Usually that's not an issue with development installs,
though, because you can put files in a place in the relative source tree
that makes sense, and then you don't need to install them into the python
location - develop installs are nicest when they do nothing more than put
in a .pth link to a package, and maybe add some entry points. Copying
files is where development installs get incredibly hairy and annoying.

EXACTLY! That's why I think package_data is the best approach to including
data files :-)

I'm not discouraging use of setup.py - I highly recommend it. I just

especially hate this snag with it.

me too.

- CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Barker]

Post by Skip Montanaro
With a bit of reading I think I determined how to specify data files

there are WAY too many ways to do it :-(

I jsut put togther these notes for my python class:

https://uwpce-pythoncert.github.io/PythonCertDevel/modules/Packaging.html#dealing-with-data-files

writing that, I played a bit with the options.

I recommend using packge_data, rather than data_files. That way, they get
installed inside the package, so much easier to bundle with with conda,
etc, and also the path of the data files relative to your code doesn't
change depending on whether the package is installed or not.

I'm actually a bit confused as to what "data_files" is for -- it puts them
outside your package, but still inside the Python install -- what's the
point of that????

(maybe if you have multiple packages that need to access the same data? but
that sure seem fragile to me!)

Also, I'm not sure what happens with "data_files" in a --user install --
could get ugly!

-CHB

Post by Skip Montanaro
setuptools.setup(
...
data_files=[('relative/dir', ['data/mumble.ini'])],
...
)
When I build my conda package, I find mumble.ini here (using python2.7
lib/python2.7/site-packages/relative/dir/mumble.ini
bin/run-my-service.sh
I think that's an odd place to put a data file, especially one which
(in my case) names a config file used by my start script installed in
"bin" (at the same level as "lib"). Is that common practice? Is there
a better way to do this?
Thanks,
Skip Montanaro
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/
continuum.io/d/msgid/conda/CANc-5Uzov7y9Opt4u65zfu%2B-
gqHxE-wpC97toJ1gAOW%3DnTK8GQ%40mail.gmail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Skip Montanaro]

Thanks, Chris. Will take a look over the weekend.

Skip

Post by Skip Montanaro
With a bit of reading I think I determined how to specify data files

there are WAY too many ways to do it :-(

I jsut put togther these notes for my python class:

https://uwpce-pythoncert.github.io/PythonCertDevel/modules/Packaging.html#
dealing-with-data-files

writing that, I played a bit with the options.

I recommend using packge_data, rather than data_files. That way, they get
installed inside the package, so much easier to bundle with with conda,
etc, and also the path of the data files relative to your code doesn't
change depending on whether the package is installed or not.

I'm actually a bit confused as to what "data_files" is for -- it puts them
outside your package, but still inside the Python install -- what's the
point of that????

(maybe if you have multiple packages that need to access the same data? but
that sure seem fragile to me!)

Also, I'm not sure what happens with "data_files" in a --user install --
could get ugly!

-CHB

Post by Skip Montanaro
setuptools.setup(
...
data_files=[('relative/dir', ['data/mumble.ini'])],
...
)
When I build my conda package, I find mumble.ini here (using python2.7
lib/python2.7/site-packages/relative/dir/mumble.ini
bin/run-my-service.sh
I think that's an odd place to put a data file, especially one which
(in my case) names a config file used by my start script installed in
"bin" (at the same level as "lib"). Is that common practice? Is there
a better way to do this?
Thanks,
Skip Montanaro
--
You received this message because you are subscribed to the Google Groups
"conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/co
ntinuum.io/d/msgid/conda/CANc-5Uzov7y9Opt4u65zfu%2B-gqHxE-
wpC97toJ1gAOW%3DnTK8GQ%40mail.gmail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE
<https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g>
(206)
526-6329 fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov