pkgutil, pkg_resource and Python 3.0 name space packages
Hello! We are discussing name space packages on the stdlib reorg list. For Python 3.0 we plan to organize the packages by purpose, e.g. put all database related packages like sqlite and shelve in a 'databases' name space. Of course we want to have the name spaces extensible by 3rd party software. The Python world as currently two ways to create extensible name space packages: pkgutil and pkg_resource. pkgutil is part of Python 2.5. pkg_resource is scheduled for Python 2.6 and 3.0 in PEP 365 [1]. The PEP hasn't been accepted yet. Questions: * PJE: Is pkg_resource ready for Python 2.6 and 3.0? * GvR: Are you going to accept Phillip's PEP? * PJE: Does pkg_resource have an easy way to overwrite a package in a name space package? E.g. an user wants to overwrite Python's databases.sqlite with a newer version of sqlite. Can he simply do it by inserting a package into sys.path before Lib/ ? Christian [1] http://www.python.org/dev/peps/pep-0365/
On Jan 6, 2008 8:33 AM, Christian Heimes <lists@cheimes.de> wrote:
* PJE: Does pkg_resource have an easy way to overwrite a package in a name space package? E.g. an user wants to overwrite Python's databases.sqlite with a newer version of sqlite. Can he simply do it by inserting a package into sys.path before Lib/ ?
Do we really want to encourage this? Wouldn't that just introduce more pyxml-like nightmares? I've been bitten way too many times by pyxml overwriting the regular xml package and causing version incompatibilities. I'd hate for this kind of thing to become common practice. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy
Steven Bethard wrote:
Do we really want to encourage this? Wouldn't that just introduce more pyxml-like nightmares? I've been bitten way too many times by pyxml overwriting the regular xml package and causing version incompatibilities. I'd hate for this kind of thing to become common practice.
I like to give 3rd party software a chance to *extend* a name space package like xml rather then to overwrite it. As far as I understand your problem pyxml is overwriting the name space and claiming it for itself rather than extending it. Christian
On Jan 6, 2008 11:34 AM, Christian Heimes <lists@cheimes.de> wrote:
Steven Bethard wrote:
Do we really want to encourage this? Wouldn't that just introduce more pyxml-like nightmares? I've been bitten way too many times by pyxml overwriting the regular xml package and causing version incompatibilities. I'd hate for this kind of thing to become common practice.
I like to give 3rd party software a chance to *extend* a name space package like xml rather then to overwrite it. As far as I understand your problem pyxml is overwriting the name space and claiming it for itself rather than extending it.
The most recent problem was that pyxml installs a different version of pyexpat so that ``xml.parsers.pyexpat`` != ``pyexpat``. This causes problems with mod_python: http://www.dscpl.com.au/wiki/ModPython/Articles/ExpatCausingApacheCrash What concerned me was your comment: E.g. an user wants to overwrite Python's databases.sqlite with a newer version of sqlite Maybe the situation is different here, but having someone installing a different version of sqlite behind my back makes me nervous. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy
On 06/01/2008, Steven Bethard <steven.bethard@gmail.com> wrote:
What concerned me was your comment:
E.g. an user wants to overwrite Python's databases.sqlite with a newer version of sqlite
Maybe the situation is different here, but having someone installing a different version of sqlite behind my back makes me nervous.
Yes, that concerned me. Whether possible or not, I see this as bad practice in general. On the other hand, allowing cx_Oracle to register itself as databases.cx_Oracle - which is the basic "namespace" functionality - is something I feel is important. (I have no idea if the cx_Oracle author would want to do this, but I think the option should be available). Paul.
At 07:34 PM 1/6/2008 +0100, Christian Heimes wrote:
Steven Bethard wrote:
Do we really want to encourage this? Wouldn't that just introduce more pyxml-like nightmares? I've been bitten way too many times by pyxml overwriting the regular xml package and causing version incompatibilities. I'd hate for this kind of thing to become common practice.
I like to give 3rd party software a chance to *extend* a name space package like xml rather then to overwrite it. As far as I understand your problem pyxml is overwriting the name space and claiming it for itself rather than extending it.
Indeed. It should also be noted that namespace packages are actually a very mature technology at this point. Before setuptools, pkgutil already supported them, from the time of 2.3's release.
At 04:33 PM 1/6/2008 +0100, Christian Heimes wrote:
Hello!
We are discussing name space packages on the stdlib reorg list. For Python 3.0 we plan to organize the packages by purpose, e.g. put all database related packages like sqlite and shelve in a 'databases' name space.
Of course we want to have the name spaces extensible by 3rd party software. The Python world as currently two ways to create extensible name space packages: pkgutil and pkg_resource.
pkgutil is part of Python 2.5. pkg_resource is scheduled for Python 2.6 and 3.0 in PEP 365 [1]. The PEP hasn't been accepted yet.
Questions:
* PJE: Is pkg_resource ready for Python 2.6 and 3.0?
The extra feature proposed in PEP 365 isn't done yet. Without the PEP acceptance, I didn't feel the need to rush on finishing it. Apart from that, the pkg_resources module has been pretty darn stable.
* GvR: Are you going to accept Phillip's PEP?
* PJE: Does pkg_resource have an easy way to overwrite a package in a name space package?
OverRIDE, yes; overWRITE, no.
E.g. an user wants to overwrite Python's databases.sqlite with a newer version of sqlite. Can he simply do it by inserting a package into sys.path before Lib/ ?
As long as the 'databases' package hasn't been imported or had its namespace declared yet, yes. (I'm making the assumption, of course, that all of the namespace package requirements have been met: i.e., each package has a database/__init__.py that contains a namespace declaration and nothing else.)
On 2008-01-06 16:33, Christian Heimes wrote:
Hello!
We are discussing name space packages on the stdlib reorg list. For Python 3.0 we plan to organize the packages by purpose, e.g. put all database related packages like sqlite and shelve in a 'databases' name space.
Regardless of whether this would really be helpful or not (I used to be in favor of this some time ago, but then realized that this doesn't really buy you anything much), please be aware that by using new generic top-level names you are likely going to get into conflict with existing applications. 'databases' will likely work since most apps will use 'database' as module or package name.
Of course we want to have the name spaces extensible by 3rd party software.
This is not a good idea. The standard lib should not be extensible by 3rd party packages, instead they should use their own top-level package and deal with whatever namespace extension mechanism is used in that package.
The Python world as currently two ways to create extensible name space packages: pkgutil and pkg_resource.
I don't think pkg_resources should go into Python 2.6 as is - for the reasons I've stated last year when this question came up. I also don't like the import mechanism hackery that's being used in setuptools to get namespace packages working. IMHO, it would be a lot cleaner to extend the existing import mechanism to support search continuation, ie. have the import find mechanism continue the search for a sub-package if it doesn't find the module in the first instance of a top-level package directory. E.g. say you want to import 'a.c' and you have directories on you sys.path: libs1/a/ libs1/a/__init__.py libs1/a/b.py libs2/a/ libs2/a/__init__.py libs2/a/c.py the import mechanism would look in libs1/a/, see that c.py is not available, then continue the search and find libs2/a/c.py. This is a sketch, but I think you get the idea. Next, we add a per-user site-packages directory to the standard sys.path, and then we could get rid of most of the setuptools import and sys.path hackery, making it a lot cleaner. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
On Jan 7, 2008, at 7:48 AM, M.-A. Lemburg wrote:
Next, we add a per-user site-packages directory to the standard sys.path, and then we could get rid of most of the setuptools import and sys.path hackery, making it a lot cleaner.
PYTHONPATH already provides this functionality. I see no need to duplicate that. -Fred -- Fred Drake <fdrake at acm.org>
On 2008-01-07 14:57, Fred Drake wrote:
On Jan 7, 2008, at 7:48 AM, M.-A. Lemburg wrote:
Next, we add a per-user site-packages directory to the standard sys.path, and then we could get rid of most of the setuptools import and sys.path hackery, making it a lot cleaner.
PYTHONPATH already provides this functionality. I see no need to duplicate that.
Agreed, but one of the main arguments for all the .pth file hackery in setuptools is that having to change PYTHONPATH in order to enable user installations of packages is too hard for the typical user. We could easily resolve that issue, if we add a per-user site-packages dir to sys.path in site.py (this is already done for Macs). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2008, at 9:01 AM, M.-A. Lemburg wrote:
We could easily resolve that issue, if we add a per-user site-packages dir to sys.path in site.py (this is already done for Macs).
+1. I've advocated that for years. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR4I4EHEjvBPtnXfVAQLpPwQAizIj/FBCc9tTNVoPRTghv37WVL0wAk8Z WtmDVpag8H/j7uNKKwMd7Ld0HdVMAwpDetdGPklDnhMDqD/jY5Da2551uKgawFnZ 57WfY5C/VvYbI8jofDEJxb2bEJyG2QILnqut+D8/9zU+z/G1ubL+jgTY03F7a71O JQAaGa6DxNU= =S5C/ -----END PGP SIGNATURE-----
On Jan 7, 2008 6:32 AM, Barry Warsaw <barry@python.org> wrote:
On Jan 7, 2008, at 9:01 AM, M.-A. Lemburg wrote:
We could easily resolve that issue, if we add a per-user site-packages dir to sys.path in site.py (this is already done for Macs).
+1. I've advocated that for years.
I'm not sure what this buys given that you can do this using PYTHONPATH anyway, but because of that I also can't be against it. +0 from me. Patches for 2.6 gratefully accepted. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2008, at 10:12 AM, Guido van Rossum wrote:
On Jan 7, 2008 6:32 AM, Barry Warsaw <barry@python.org> wrote:
On Jan 7, 2008, at 9:01 AM, M.-A. Lemburg wrote:
We could easily resolve that issue, if we add a per-user site- packages dir to sys.path in site.py (this is already done for Macs).
+1. I've advocated that for years.
I'm not sure what this buys given that you can do this using PYTHONPATH anyway, but because of that I also can't be against it. +0 from me. Patches for 2.6 gratefully accepted.
I think it's PEP-worthy too, just so that the semantics get nailed down. Here's a strawman proto-quasi-pre-PEP. Python automatically adds ~/.python/site-packages to sys.path; this is added /before/ the system site-packages file. An open question is whether it needs to go at the front of the list. It should definitely be searched before the system site-packages. Python treats ~/.python/site-packages the same as the system site- packages, w.r.t. .pth files, etc. Open question: should we add yet another environment variable to control this? It's pretty typical for apps to expose such a thing so that the base directory (e.g. ~/.python) can be moved. I think that's all that's needed. It would make playing with easy_install/setuptools nicer to have this. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR4JSJ3EjvBPtnXfVAQIXfgP9HCi8PNwUbPTbeaf7JPHLTkguYwUAyLH2 vD31w76fnh3pChIZSY9CtP51qRmB22DChSQxaagLmnl9FXjdS1dmXOu7oT7lfj2z avTptyJ2MB8kFuGLK2In/mjbWxPUgAd19sbm4jtE5b3nauOBVyZh23TxFvH5uRdD bUaqRNib3vE= =P3p4 -----END PGP SIGNATURE-----
On 2008-01-07 17:24, Barry Warsaw wrote:
On Jan 7, 2008, at 10:12 AM, Guido van Rossum wrote:
On Jan 7, 2008 6:32 AM, Barry Warsaw <barry@python.org> wrote:
On Jan 7, 2008, at 9:01 AM, M.-A. Lemburg wrote:
We could easily resolve that issue, if we add a per-user site-packages dir to sys.path in site.py (this is already done for Macs).
+1. I've advocated that for years.
I'm not sure what this buys given that you can do this using PYTHONPATH anyway, but because of that I also can't be against it. +0 from me. Patches for 2.6 gratefully accepted.
I think it's PEP-worthy too, just so that the semantics get nailed down. Here's a strawman proto-quasi-pre-PEP.
Python automatically adds ~/.python/site-packages to sys.path; this is added /before/ the system site-packages file. An open question is whether it needs to go at the front of the list. It should definitely be searched before the system site-packages.
Python treats ~/.python/site-packages the same as the system site-packages, w.r.t. .pth files, etc.
Open question: should we add yet another environment variable to control this? It's pretty typical for apps to expose such a thing so that the base directory (e.g. ~/.python) can be moved.
I'd suggest to make the "~/.python" part configurable by an env var, e.g. PYTHONRESOURCES. Perhaps we could use that directory for other Python-related resources as well, e.g. an optional sys.path lookup cache (pickled dictionary of known package/module file locations to reduces Python startup time).
I think that's all that's needed. It would make playing with easy_install/setuptools nicer to have this.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2008, at 11:30 AM, M.-A. Lemburg wrote:
Open question: should we add yet another environment variable to control this? It's pretty typical for apps to expose such a thing so that the base directory (e.g. ~/.python) can be moved.
I'd suggest to make the "~/.python" part configurable by an env var, e.g. PYTHONRESOURCES.
Since we can't use PYTHONOHOME, this works for me.
Perhaps we could use that directory for other Python-related resources as well, e.g. an optional sys.path lookup cache (pickled dictionary of known package/module file locations to reduces Python startup time).
Sure, why not? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR4KTn3EjvBPtnXfVAQLQlAP/R887WX0k3VNPoiyVs3qerYBj0XQyP4DT leyFWInEamtgk1+5hL+Vu180k+EFcjEQ8d2yet3sMMVUZ4piFHEd6SdNJantVKrl YBVTHkhEBNX2qMFxYmyTwzvjaMttbIn5w9TuEG4PnNiYQv4E4HlZ9HOkRY9YuNDh UMr4e4DCZtw= =PI+B -----END PGP SIGNATURE-----
At 11:24 AM 1/7/2008 -0500, Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jan 7, 2008, at 10:12 AM, Guido van Rossum wrote:
On Jan 7, 2008 6:32 AM, Barry Warsaw <barry@python.org> wrote:
On Jan 7, 2008, at 9:01 AM, M.-A. Lemburg wrote:
We could easily resolve that issue, if we add a per-user site- packages dir to sys.path in site.py (this is already done for Macs).
+1. I've advocated that for years.
I'm not sure what this buys given that you can do this using PYTHONPATH anyway, but because of that I also can't be against it. +0 from me. Patches for 2.6 gratefully accepted.
I think it's PEP-worthy too, just so that the semantics get nailed down. Here's a strawman proto-quasi-pre-PEP.
Python automatically adds ~/.python/site-packages to sys.path; this is added /before/ the system site-packages file. An open question is whether it needs to go at the front of the list. It should definitely be searched before the system site-packages.
What about including the Python version in the directory name? C Extensions may not work correctly across versions, and bytecode will get recompiled a lot if you're using multiple versions. Also, if this is a 2.6/3.0 change, it's likely that the *source* won't be compatible across versions either. :)
Python treats ~/.python/site-packages the same as the system site- packages, w.r.t. .pth files, etc.
Open question: should we add yet another environment variable to control this? It's pretty typical for apps to expose such a thing so that the base directory (e.g. ~/.python) can be moved.
I think that's all that's needed. It would make playing with easy_install/setuptools nicer to have this.
Assuming that this is a true "site" directory (i.e., .pth files are recognized), then yes.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2008, at 11:37 AM, Phillip J. Eby wrote:
Python automatically adds ~/.python/site-packages to sys.path; this is added /before/ the system site-packages file. An open question is whether it needs to go at the front of the list. It should definitely be searched before the system site-packages.
What about including the Python version in the directory name? C Extensions may not work correctly across versions, and bytecode will get recompiled a lot if you're using multiple versions. Also, if this is a 2.6/3.0 change, it's likely that the *source* won't be compatible across versions either. :)
D'oh, yes of course. So make that: ~/.python/lib/pythonX.Y/site-packages
Python treats ~/.python/site-packages the same as the system site- packages, w.r.t. .pth files, etc.
Open question: should we add yet another environment variable to control this? It's pretty typical for apps to expose such a thing so that the base directory (e.g. ~/.python) can be moved.
I think that's all that's needed. It would make playing with easy_install/setuptools nicer to have this.
Assuming that this is a true "site" directory (i.e., .pth files are recognized), then yes.
IMO, it should be a true site directory. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR4KTN3EjvBPtnXfVAQI6twP9HUQ23I0KCGZy4CC9sA6vvM5xYfmEgKQb 7H9AP84nSqaaX0iyONmRqlPahzHaEkjkoMW68EA3AIkXagf72sNbyBMO1p7I7ZQ6 5X6dR78o8w+NywO6sgdgxqQq3ikXNEEAy2EICLamT94Um1QxR7OV7SlihdpDs20w MujSDYAr9CQ= =el6S -----END PGP SIGNATURE-----
* Barry Warsaw <barry@python.org> [2008-01-07 16:01:42 -0500]:
On Jan 7, 2008, at 11:37 AM, Phillip J. Eby wrote:
Python automatically adds ~/.python/site-packages to sys.path; this is added /before/ the system site-packages file. An open question is whether it needs to go at the front of the list. It should definitely be searched before the system site-packages.
What about including the Python version in the directory name? C Extensions may not work correctly across versions, and bytecode will get recompiled a lot if you're using multiple versions. Also, if this is a 2.6/3.0 change, it's likely that the *source* won't be compatible across versions either. :)
D'oh, yes of course. So make that:
~/.python/lib/pythonX.Y/site-packages
In that case how about: ~/.local/lib/pythonX.Y/site-packages or: ~/local/lib/pythonX.Y/site-packages I believe both of these locations are already in use by various systems / people, so it would fit in better with existing practice.
IMO, it should be a true site directory.
This would be ideal. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On 07/01/2008, Tristan Seligmann <mithrandi-python-dev@mithrandi.za.net> wrote:
D'oh, yes of course. So make that:
~/.python/lib/pythonX.Y/site-packages
In that case how about:
~/.local/lib/pythonX.Y/site-packages
or:
~/local/lib/pythonX.Y/site-packages
What would be used on Windows? It's likely to be of marginal use on Windows, but an appropriate equivalent should be defined. Possibly just replace ~ with %USERPROFILE%. I'd argue against anything under %APPDATA% as that directory is hidden. Also, should bdist_wininst/bdist_msi installers be updated to offer the option of installing to this directory? What about python setup.py install (add a --user flag, for example)? Paul.
Paul Moore wrote:
On 07/01/2008, Tristan Seligmann <mithrandi-python-dev@mithrandi.za.net> wrote:
D'oh, yes of course. So make that:
~/.python/lib/pythonX.Y/site-packages
In that case how about:
~/.local/lib/pythonX.Y/site-packages
or:
~/local/lib/pythonX.Y/site-packages
What would be used on Windows? It's likely to be of marginal use on Windows,
Could be very useful. A lot of machines are used with multiple users.
but an appropriate equivalent should be defined. Possibly just replace ~ with %USERPROFILE%. I'd argue against anything under %APPDATA% as that directory is hidden.
Replacing ~ with %USERPROFILE% sounds like the right thing to do.
Also, should bdist_wininst/bdist_msi installers be updated to offer the option of installing to this directory? What about python setup.py install (add a --user flag, for example)?
The installers should give the user the choice at install time (preferably). A '--user' flag would also be useful (IMHO). Michael Foord
Paul. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
Paul Moore wrote:
What would be used on Windows? It's likely to be of marginal use on Windows, but an appropriate equivalent should be defined. Possibly just replace ~ with %USERPROFILE%. I'd argue against anything under %APPDATA% as that directory is hidden.
No, we shouldn't mess with the profile root directory on Windows. The data should either be installed under "Application Data" or under "My Documents". You are right, the appdata directory is hidden by the installer could add a link to Application Data\Python\python2.x\ to the start menu. I also don't like the idea to replace ~ with another string. Every OS is probably using a different path (~/.python/ on Linux, ~/Library/Python on Mac and APPDATA\Python on Windows). The distutils package should gain two methods: global_sitedir() and local_sitedir([username]).
Also, should bdist_wininst/bdist_msi installers be updated to offer the option of installing to this directory? What about python setup.py install (add a --user flag, for example)?
Sounds good to me. Apropos My Documents and other special directories on Windows. Python doesn't have an API to get the directories from the registry. Is somebody interested in having a module for the task? I've some code for the job on disk. Christian
On 08/01/2008, Christian Heimes <lists@cheimes.de> wrote:
Paul Moore wrote:
What would be used on Windows? It's likely to be of marginal use on Windows, but an appropriate equivalent should be defined. Possibly just replace ~ with %USERPROFILE%. I'd argue against anything under %APPDATA% as that directory is hidden.
No, we shouldn't mess with the profile root directory on Windows. The data should either be installed under "Application Data" or under "My Documents". You are right, the appdata directory is hidden by the installer could add a link to Application Data\Python\python2.x\ to the start menu.
Not My Documents, please! That's for documents, not configuration. %USERPROFILE% is actually where most other applications put stuff. The alternative would be %HOMEDRIVE%%HOMEPATH% which is what os.path.expanduser uses.
Apropos My Documents and other special directories on Windows. Python doesn't have an API to get the directories from the registry. Is somebody interested in having a module for the task? I've some code for the job on disk.
It would probably be a good idea to have it in the core, although I'm sure it's in pywin32, and anyone coding Python on Windows will have that. Personally, I've no code that would benefit from this, so I'd be +0 on theoretical grounds only. Paul.
Paul Moore wrote:
Not My Documents, please! That's for documents, not configuration. %USERPROFILE% is actually where most other applications put stuff. The alternative would be %HOMEDRIVE%%HOMEPATH% which is what os.path.expanduser uses.
On mys system only one application has put configuration data directly into USERPROFILE. It's Python's IDLE and I don't like it. It should store its configuration under APPDATA. I also don't agree that Python extensions are configuration data. They are code, maybe plugins and the files are user editable content. Ms products like Visual Studio store files like them in My Documents.
It would probably be a good idea to have it in the core, although I'm sure it's in pywin32, and anyone coding Python on Windows will have that. Personally, I've no code that would benefit from this, so I'd be +0 on theoretical grounds only.
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments(). Christian
-On [20080108 17:07], Christian Heimes (lists@cheimes.de) wrote:
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments().
Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users. So you get: C:\Users\<name>\AppData\Local\ (former Local Settings\Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos) -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ Vae victis!
Jeroen Ruigrok van der Werven wrote:
-On [20080108 17:07], Christian Heimes (lists@cheimes.de) wrote:
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments().
Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users.
So you get:
C:\Users\<name>\AppData\Local\ (former Local Settings\Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos)
Somewhat off topic, but hooray, it looks like someone at MS rediscovered the command line and why "Long and Wordy Names with lots of spaces" are really annoying. Easier access to the application data directory is a good thing too. I guess moving to Vista wouldn't be all bad then ;) To get marginally back on topic, I would actually prefer to have a function like os.gethomedir() that takes on optional user name. (I don't want it enough to write a patch to make it happen though) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
Jeroen Ruigrok van der Werven wrote:
-On [20080108 17:07], Christian Heimes (lists@cheimes.de) wrote:
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments().
Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users.
So you get:
C:\Users\<name>\AppData\Local\ (former Local Settings\Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos)
Note today's Coding Horror blog entry: "Don't Pollute User Space" http://www.codinghorror.com/blog/archives/001032.html Keep your dirty, filthy paws out of my personal user space! Take a look in your Documents folder right now. Go ahead. Look. Do you see any files or folders in there that you personally did not create? If so, you've been victimized. Applications should never create or modify anything in your documents folder without your permission. {snip...] If applications need to store shared files, that's what the \AppData and \Application Data folders are for. Sentiments I agree with... Michael http://www.manning.com/foord
Somewhat off topic, but hooray, it looks like someone at MS rediscovered the command line and why "Long and Wordy Names with lots of spaces" are really annoying. Easier access to the application data directory is a good thing too.
I guess moving to Vista wouldn't be all bad then ;)
To get marginally back on topic, I would actually prefer to have a function like os.gethomedir() that takes on optional user name. (I don't want it enough to write a patch to make it happen though)
Cheers, Nick.
On 09/01/2008, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Note today's Coding Horror blog entry: "Don't Pollute User Space"
http://www.codinghorror.com/blog/archives/001032.html
Keep your dirty, filthy paws out of my personal user space!
:-) Absolutely [...]
If applications need to store shared files, that's what the \AppData and \Application Data folders are for.
Sentiments I agree with...
Yes, with the one proviso that Windows (XP, at least, maybe Vista is actually better in this regard) makes it extremely difficult for the user to manually do anything in these directories, so that I'd say this is only appropriate for fully application-maintained files. As far as I know, Windows lacks any really sensible place to store application configuration data which is meant to be user edited on occasion (e.g. rc/ini files, startup scripts, etc). That's often what I see ending up in %USERPROFILE% - even though as Christian points out, it's "officially" wrong. Paul.
Paul Moore wrote:
On 09/01/2008, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Note today's Coding Horror blog entry: "Don't Pollute User Space"
http://www.codinghorror.com/blog/archives/001032.html
Keep your dirty, filthy paws out of my personal user space!
:-) Absolutely
[...]
If applications need to store shared files, that's what the \AppData and \Application Data folders are for.
Sentiments I agree with...
Yes, with the one proviso that Windows (XP, at least, maybe Vista is actually better in this regard) makes it extremely difficult for the user to manually do anything in these directories,
Only because Windows XP uses a stupidly long path with spaces in it. It's not actually *hard* to navigate manually to these directories. If the windows installers created by the 'bdist' commands allow you to automatically put stuff there then it shouldn't be too much of a problem.
so that I'd say this is only appropriate for fully application-maintained files. As far as I know, Windows lacks any really sensible place to store application configuration data which is meant to be user edited on occasion (e.g. rc/ini files, startup scripts, etc).
What user editing is *meant* to be done with extension modules installed into a site directory. Programmer editing maybe... :-) Michael Foord
That's often what I see ending up in %USERPROFILE% - even though as Christian points out, it's "officially" wrong.
Paul.
On 09/01/2008, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Only because Windows XP uses a stupidly long path with spaces in it. It's not actually *hard* to navigate manually to these directories.
The directories are also hidden. That does make it hard to navigate there. I know you can un-hide hidden files, but I view the hidden attribute as useful - just badly misused in this case, unless you assume that these directories are intended to be left alone by the user.
What user editing is *meant* to be done with extension modules installed into a site directory. Programmer editing maybe... :-)
Sorry, I had drifted slightly off topic here. I don't have a problem with user-specific extensions going in appdata, on the presumption that we're talking code only (and excluding something obscure like distutils' distutils.cfg file). Paul.
Paul Moore wrote:
The directories are also hidden. That does make it hard to navigate there. I know you can un-hide hidden files, but I view the hidden attribute as useful - just badly misused in this case, unless you assume that these directories are intended to be left alone by the user.
It's not an issue for experienced users. For the rest we can put a link in the start menu under Python 2.5 which opens a new explorer with the user package directory. Christian
On 09/01/2008, Christian Heimes <lists@cheimes.de> wrote:
It's not an issue for experienced users. For the rest we can put a link in the start menu under Python 2.5 which opens a new explorer with the user package directory.
Um, I'm an experienced user and it's an issue for me... The problem isn't so much knowing where the directory is, as that tools (quite rightly) ignore (ie, hide!) hidden directories. For example, 4NT (my command shell) doesn't auto-complete hidden directory names by default, many tools' wildcard matching ignores them (always, not just by default). Blast, I said I wasn't going to start a big flamewar over Windows behaviour. I haven't seen anyone propose that something intended to be edited by a user be stored in a hidden directory (appdata) on Windows. As long as that is the case, I don't care. If you are suggesting that a file intended to be viewed/edited by a user manually should go in AppData, then please be explicit. We can then argue the concrete issues, rather than just theoretical principles. Paul.
Paul Moore wrote:
If you are suggesting that a file intended to be viewed/edited by a user manually should go in AppData, then please be explicit. We can then argue the concrete issues, rather than just theoretical principles.
I'm frustrated as well. Neither AppData nor MyDocuments fulfill our requirements. I don't argue for AppData, I only argue against UserProfile as the base directory for a user package directory. I'm open for any suggestion which doesn't violate MS' style guides. (I don't want to imply that I like the style guide but we don't make the rules :( ) Christian
On 09/01/2008, Christian Heimes <lists@cheimes.de> wrote:
Paul Moore wrote:
If you are suggesting that a file intended to be viewed/edited by a user manually should go in AppData, then please be explicit. We can then argue the concrete issues, rather than just theoretical principles.
I'm frustrated as well. Neither AppData nor MyDocuments fulfill our requirements.
For what? If you're talking about a per-user site-packages directory, I have no problem with AppData (as Michael said, there's no intention that users be able to edit the files). Of course, modules/packages/extensions don't really qualify as application *data*, so it is a somewhat odd usage certainly, but it's not disastrous. Windows doesn't really have a concept of a per-user application code directory. Come to that, if I use a bdist_wininst installer to install a package in my personal site packages, then I log off and you log on, would you expect to see my package when you look in add/remove programs? Would you expect to be able to uninstall it? Would you expect to be able to install a personal copy of your own? What would we see in add/remove programs *then*?? No matter how you cut it, Windows isn't designed for per-user installable programs. Maybe a per-user site-packages just isn't appropriate on Windows. Paul. PS This is mostly theoretical for me, as I don't have a personal need for a per-user site packages directory on any of the machines I use.
Paul Moore wrote: [...]
No matter how you cut it, Windows isn't designed for per-user installable programs. Maybe a per-user site-packages just isn't appropriate on Windows.
This reminds me of the early days of Microsoft Terminal Service (read: "X Window done wrong fifteen years later"), when various applications had to be modified to stop saving per-user data in global locations. It underlines the fact that essentially Windows is still betrayed by its original conception as a single-user system. The idea that users would /program their own computers/ was totally alien to the Windows mindset. looking-at-Vista-and-thinking-very-hard-ly y'rs - steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On 09/01/2008, Steve Holden <steve@holdenweb.com> wrote:
The idea that users would /program their own computers/ was totally alien to the Windows mindset.
Actually, the alien idea is that more than one person would use the same (Windows) computer. Not surprising as these were *personal* computers. It's Windows as a server OS that's the bizarre idea... Paul.
On Jan 9, 2008, at 1:48 AM, Jeroen Ruigrok van der Werven wrote:
-On [20080108 17:07], Christian Heimes (lists@cheimes.de) wrote:
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments().
Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users.
So you get:
C:\Users\<name>\AppData\Local\ (former Local Settings \Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos)
yay, next up posix support.... *dreams* ~ro
-- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ Vae victis! _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/reed% 40reedobrien.com
Reed O'Brien wrote:
On Jan 9, 2008, at 1:48 AM, Jeroen Ruigrok van der Werven wrote:
-On [20080108 17:07], Christian Heimes (lists@cheimes.de) wrote:
Python's _winreg module and pywin32 expose several functions to get the paths from the registry but I don't think it has a simple function like get_mydocuments(). Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users.
So you get: C:\Users\<name>\AppData\Local\ (former Local Settings\Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos) yay, next up posix support....
I suspect that the whole thing was done to make sure that developers of applications could: A: cope with stupidly long path names. V: cope with spaces in path names. I bet they never intended to keep the huge names, just to make you cope with them. --Scott David Daniels Scott.Daniels@Acm.Org
Jeroen Ruigrok van der Werven wrote:
Careful with the name though. Microsoft Windows Vista did away with 'My Documents & Settings'. It is now C:\Users.
So you get:
C:\Users\<name>\AppData\Local\ (former Local Settings\Application Data) C:\Users\<name>\AppData\Roaming\ (former Application Data) C:\Users\<name>\Documents (former My Documents) C:\Users\<name>\Music (former My Music) C:\Users\<name>\Pictures (former My Pictures) C:\Users\<name>\Videos (former My Videos)
My latest version uses the SHGetFolderPathW() function from ShlObj.h. It's the preferred way to get shell folder paths like CLSID_PERSONAL (my documents). It's compatible with 2000 and newer (maybe even older, but we don't support ME, NT4 or older) and works on Vista, too. Christian
Paul Moore wrote:
Not My Documents, please! That's for documents, not configuration. %USERPROFILE% is actually where most other applications put stuff. The alternative would be %HOMEDRIVE%%HOMEPATH% which is what os.path.expanduser uses.
http://msdn2.microsoft.com/en-us/library/bb762494(VS.85).aspx CSIDL_PROFILE (FOLDERID_Profile) Version 5.0. The user's profile folder. A typical path is C:\Documents and Settings\username. Applications should not create files or folders at this level; they should put their data under the locations referred to by CSIDL_APPDATA or CSIDL_LOCAL_APPDATA. Christian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2008, at 5:49 PM, Tristan Seligmann wrote:
In that case how about:
~/.local/lib/pythonX.Y/site-packages
or:
~/local/lib/pythonX.Y/site-packages
I believe both of these locations are already in use by various systems / people, so it would fit in better with existing practice.
I like the former path. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBR4VoJ3EjvBPtnXfVAQK9zQP/dup8RHQha0v9ZkYee+L5/8/fhvIHBEoP MGkBxZUv79SmtnY4WWovS66NYAuKHlChFN0VSZCUO8o7WE8b52lXhgeG2xGX/pU/ MZ6D6zjxZmFhQjLLNf5gRcvchYHM+18AYQFapnuCpV048cgmJ7DQcGqznpyYRhnm SvqTeBtstNw= =HYwR -----END PGP SIGNATURE-----
At 03:01 PM 1/7/2008 +0100, M.-A. Lemburg wrote:
On 2008-01-07 14:57, Fred Drake wrote:
On Jan 7, 2008, at 7:48 AM, M.-A. Lemburg wrote:
Next, we add a per-user site-packages directory to the standard sys.path, and then we could get rid of most of the setuptools import and sys.path hackery, making it a lot cleaner.
PYTHONPATH already provides this functionality. I see no need to duplicate that.
Agreed, but one of the main arguments for all the .pth file hackery in setuptools is that having to change PYTHONPATH in order to enable user installations of packages is too hard for the typical user.
We could easily resolve that issue, if we add a per-user site-packages dir to sys.path in site.py (this is already done for Macs).
Actually, neither PYTHONPATH nor your proposal solve all of the problems that .pth files do. To date, nobody has proposed any real substitute for them.
At 01:48 PM 1/7/2008 +0100, M.-A. Lemburg wrote:
I also don't like the import mechanism hackery that's being used in setuptools to get namespace packages working.
I believe you're mistaken: there is no import mechanism "hackery" in pkg_resources. (__path__ is a documented *hook*, not a hack, and it's the only import-related hook that pkg_resources uses). And, if you don't like namespace packages, perhaps you should be campaigning for this to be removed: http://python.org/doc/2.4.1/lib/module-pkgutil.html pkg_resources only updates the routine provided in pkgutil to be a bit more automatic, and to better support PEP 302 and zipfile importing.
participants (15)
-
Barry Warsaw -
Christian Heimes -
Fred Drake -
Guido van Rossum -
Jeroen Ruigrok van der Werven -
M.-A. Lemburg -
Michael Foord -
Nick Coghlan -
Paul Moore -
Phillip J. Eby -
Reed O'Brien -
Scott David Daniels -
Steve Holden -
Steven Bethard -
Tristan Seligmann