On 2007-01-02 01:02, brett.cannon wrote:
Author: brett.cannon Date: Tue Jan 2 01:02:41 2007 New Revision: 53204
Added: peps/trunk/pep-3108.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: Add PEP 3108: Standard Library Reorganization.
...
+Open Issues +=========== + +Consolidate dependent modules together into a single module or package? ... +Consolidate certain modules with similar themes together in a package? +---------------------------------------------------------------------- ...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future. Together with absolute imports this also improves the readability of modules since it becomes immediately clear where the imported code is coming from. Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
+Packages are often used to group together modules that have a similar +theme but do not have any direct relationship or dependency upon each +other. For Python 3.0 obvious groupings could be done since renaming +of various modules is already occurring. + +* collections + + heapq + + Queue + + sets + + UserDist + + UserList + + What to do with UserString? + - Have a package for Python implementations of built-in types + instead of putting the User* modules into 'collections'? +* mac + + Various Mac-specific modules. + + Same can be done for other platform-specific code. +* Profiling + + cProfile + + profile + + hotshot + + pstats +* email + + mailbox + + mhlib +* Databases + + anydbm + + dbhash + + dbm + + bsddb + + dumbdbm + + gdbm + + whichdb +* Audio + + aifc + + audioop + + chunk + + ossaudiodev + + sunau + + wave + + winsound +* Servers + + BaseHTTPServer + + CGIHTTPServer + + DocXMLRPCServer + + SimpleHTTPServer + + SimpleXMLRPCServer + + SocketServer
The package names should probably be converted to lower-case to follow PEP 8. Thanks and Happy New Year, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 02 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
On 2007-01-02 01:02, brett.cannon wrote:
Author: brett.cannon Date: Tue Jan 2 01:02:41 2007 New Revision: 53204
Added: peps/trunk/pep-3108.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: Add PEP 3108: Standard Library Reorganization.
...
+Open Issues +=========== + +Consolidate dependent modules together into a single module or package? ... +Consolidate certain modules with similar themes together in a package? +---------------------------------------------------------------------- ...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal. Together with absolute imports this also improves the readability
of modules since it becomes immediately clear where the imported code is coming from.
Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
+Packages are often used to group together modules that have a similar +theme but do not have any direct relationship or dependency upon each +other. For Python 3.0 obvious groupings could be done since renaming +of various modules is already occurring. + +* collections + + heapq + + Queue + + sets + + UserDist + + UserList + + What to do with UserString? + - Have a package for Python implementations of built-in types + instead of putting the User* modules into 'collections'? +* mac + + Various Mac-specific modules. + + Same can be done for other platform-specific code. +* Profiling + + cProfile + + profile + + hotshot + + pstats +* email + + mailbox + + mhlib +* Databases + + anydbm + + dbhash + + dbm + + bsddb + + dumbdbm + + gdbm + + whichdb +* Audio + + aifc + + audioop + + chunk + + ossaudiodev + + sunau + + wave + + winsound +* Servers + + BaseHTTPServer + + CGIHTTPServer + + DocXMLRPCServer + + SimpleHTTPServer + + SimpleXMLRPCServer + + SocketServer
The package names should probably be converted to lower-case to follow PEP 8.
Oops, I should have clarified that was not package name suggestsions beyond 'collections'. It was just meant to act as what the type of grouping was. -Brett
On 2007-01-02 23:54, Brett Cannon wrote:
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
Author: brett.cannon Date: Tue Jan 2 01:02:41 2007 New Revision: 53204
Added: peps/trunk/pep-3108.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: Add PEP 3108: Standard Library Reorganization.
...
+Open Issues +=========== + +Consolidate dependent modules together into a single module or
On 2007-01-02 01:02, brett.cannon wrote: package?
... +Consolidate certain modules with similar themes together in a package? +---------------------------------------------------------------------- ...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal.
Even if it is shot down again, it still deserves to be documented together with the reasons for being shot down. This is a one-in-a-lifetime chance, so it would be sad if it were not taken into account. The extra effort would be minimal - the renaming would have to be done using a script anyway and adding an extra 'from py import ' prefix to the modules wouldn't really make the renaming more complicated ;-)
Together with absolute imports this also improves the readability
of modules since it becomes immediately clear where the imported code is coming from.
Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 02 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
Author: brett.cannon Date: Tue Jan 2 01:02:41 2007 New Revision: 53204
Added: peps/trunk/pep-3108.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: Add PEP 3108: Standard Library Reorganization.
...
+Open Issues +=========== + +Consolidate dependent modules together into a single module or
On 2007-01-02 01:02, brett.cannon wrote: package?
... +Consolidate certain modules with similar themes together in a
On 2007-01-02 23:54, Brett Cannon wrote: package?
+----------------------------------------------------------------------
...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal.
Even if it is shot down again, it still deserves to be documented together with the reasons for being shot down.
Aw, but that means I have to go find why Guido didn't like it. =) But yes, it should be either an open issue or rejected idea. This is a one-in-a-lifetime chance, so it would be sad if it were
not taken into account.
The extra effort would be minimal - the renaming would have to be done using a script anyway and adding an extra 'from py import ' prefix to the modules wouldn't really make the renaming more complicated ;-)
=)
Together with absolute imports this also improves the readability
of modules since it becomes immediately clear where the imported code is coming from.
Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
Good point. Could actually have the py namespace be special and use a separate path list (e.g., sys.library_path) for that specific namespace. -Brett
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
Author: brett.cannon Date: Tue Jan 2 01:02:41 2007 New Revision: 53204
Added: peps/trunk/pep-3108.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: Add PEP 3108: Standard Library Reorganization.
...
+Open Issues +=========== + +Consolidate dependent modules together into a single module or
On 2007-01-02 01:02, brett.cannon wrote: package?
... +Consolidate certain modules with similar themes together in a
On 2007-01-02 23:54, Brett Cannon wrote: package?
+----------------------------------------------------------------------
...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal.
Even if it is shot down again, it still deserves to be documented together with the reasons for being shot down.
This is a one-in-a-lifetime chance, so it would be sad if it were not taken into account.
The extra effort would be minimal - the renaming would have to be done using a script anyway and adding an extra 'from py import ' prefix to the modules wouldn't really make the renaming more complicated ;-)
I was about to start writing an open issue on this since the biggest objection from Guido I could find on this topic is http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but then it started to feel like a separate PEP to me. So I think I am going to pass on taking on this topic and let someone else tackle it in a PEP. Sorry, MAL, but I need to worry about my sanity on this one. =) -Brett
On 2007-01-03 01:42, Brett Cannon wrote:
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
+Open Issues +=========== + +Consolidate dependent modules together into a single module or package? ... +Consolidate certain modules with similar themes together in a package?
+----------------------------------------------------------------------
...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal.
Even if it is shot down again, it still deserves to be documented together with the reasons for being shot down.
This is a one-in-a-lifetime chance, so it would be sad if it were not taken into account.
The extra effort would be minimal - the renaming would have to be done using a script anyway and adding an extra 'from py import ' prefix to the modules wouldn't really make the renaming more complicated ;-)
I was about to start writing an open issue on this since the biggest objection from Guido I could find on this topic is http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but then it started to feel like a separate PEP to me. So I think I am going to pass on taking on this topic and let someone else tackle it in a PEP. Sorry, MAL, but I need to worry about my sanity on this one. =)
Oh well, it seemed like a perfect fit for the scope of PEP 3108. Guido's reply seems to suggest that he's in favor of introducing a multi-package stdlib structure: """
I'm rejecting the proposal of a single top-level package named "python".
You've written that before, but you still haven't given any explanation of why a single package would be worse than a multi-level hierarchy of modules (e.g. grouped by application space).
Because a single package doesn't have any other benefits besides getting out of the way from 3rd party developers. At least a proper hierarchy would have the other benefits of grouping. (But better make it a shallow hierarchy! remember "Flat is better than nested.") """ AFAICT, he was only objecting having a single package without any extra restructuring. Then again, the post is from 2002 - so things may have changed. There have been a couple of attempts to reorg the stdlib into packages, but AFAIR, I see, all of them were withdrawn due to the problem of finding a suitable grouping (often enough, a module would be suitable for more than just one functional package, e.g. urllib would fit "io" as well as "net") or lack of support from the developers. Now that we're discussing moving the include files into a subdirectory (for much the same reasons), I think it's time to reboot the discussion of a Python package with or without possible subpackages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 04 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
On 1/4/07, M.-A. Lemburg <mal@egenix.com> wrote:
On 2007-01-03 01:42, Brett Cannon wrote:
On 1/2/07, M.-A. Lemburg <mal@egenix.com> wrote:
+Open Issues +=========== + +Consolidate dependent modules together into a single module or package? ... +Consolidate certain modules with similar themes together in a package?
+----------------------------------------------------------------------
...
If you do follow this route, please take the chance to place the whole Python stdlib under a single package. That way we'll avoid name clashes with existing packages and modules now and in the future.
That has been suggested before (including by me) and Guido has always shot it down. That's why I left it out of this proposal.
Even if it is shot down again, it still deserves to be documented together with the reasons for being shot down.
This is a one-in-a-lifetime chance, so it would be sad if it were not taken into account.
The extra effort would be minimal - the renaming would have to be done using a script anyway and adding an extra 'from py import ' prefix to the modules wouldn't really make the renaming more complicated ;-)
I was about to start writing an open issue on this since the biggest objection from Guido I could find on this topic is http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but then it started to feel like a separate PEP to me. So I think I am going to pass on taking on this topic and let someone else tackle it in a PEP. Sorry, MAL, but I need to worry about my sanity on this one. =)
Oh well, it seemed like a perfect fit for the scope of PEP 3108.
I know, but I honestly just don't have the energy to deal with it. If you want to spear-head the discussion and help me add it to the PEP, then that's great. Guido's reply seems to suggest that he's in favor of introducing
a multi-package stdlib structure:
"""
I'm rejecting the proposal of a single top-level package named "python".
You've written that before, but you still haven't given any explanation of why a single package would be worse than a multi-level hierarchy of modules (e.g. grouped by application space).
Because a single package doesn't have any other benefits besides getting out of the way from 3rd party developers.
At least a proper hierarchy would have the other benefits of grouping. (But better make it a shallow hierarchy! remember "Flat is better than nested.") """
AFAICT, he was only objecting having a single package without any extra restructuring.
Yep. PEP 3108 does have some basic package suggestions in the Open Issues section and people seem to support them. I will be making a separate push for them on python-3000 once the whole discussion of what modules to remove has settled down. Then again, the post is from 2002 - so things may have changed. Maybe. There have been a couple of attempts to reorg the stdlib into
packages, but AFAIR, I see, all of them were withdrawn due to the problem of finding a suitable grouping (often enough, a module would be suitable for more than just one functional package, e.g. urllib would fit "io" as well as "net") or lack of support from the developers.
Yep, that's what has happened. Now that we're discussing moving the include files into
a subdirectory (for much the same reasons), I think it's time to reboot the discussion of a Python package with or without possible subpackages.
Well, perhaps other people want to show support if they like the idea? I am personally split down the middle either way. -Brett
Brett Cannon wrote: [ ... ]
Yep. PEP 3108 does have some basic package suggestions in the Open Issues section and people seem to support them. I will be making a separate push for them on python-3000 once the whole discussion of what modules to remove has settled down.
Then again, the post is from 2002 - so things may have changed.
Maybe.
There have been a couple of attempts to reorg the stdlib into packages, but AFAIR, I see, all of them were withdrawn due to the problem of finding a suitable grouping (often enough, a module would be suitable for more than just one functional package, e.g. urllib would fit "io" as well as "net") or lack of support from the developers.
Yep, that's what has happened.
I can't believe that we need to be flummoxed by the necessity of having the same package appear at two (or more!) different places in the package naming hierarchy. I suspect lack of support is more due to developers feeling there are more profitable ways to spend their time.
Now that we're discussing moving the include files into a subdirectory (for much the same reasons), I think it's time to reboot the discussion of a Python package with or without possible subpackages.
Well, perhaps other people want to show support if they like the idea? I am personally split down the middle either way.
It would be an excellent idea to clean up the standard library space. It should be possible in most cases to provide backwards-compatible implementations of the current modules (at least the pure Python ones) by doing an import * from the appropriate new-style package. Some such compatibility mechanism will be essential if the re-org is to happen in an acceptable way before Py3k. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 2, 2007, at 5:41 PM, M.-A. Lemburg wrote:
Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
Sometimes though you want to do this, as when you want your application to ensure it gets a particular version of a standard library module, regardless of the version of Python being used. And now we're back to application-specific site-packages ;). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRZrsL3EjvBPtnXfVAQKVnAQAkJBlZ0nijuD062qu1Z97WTt0To07nLEw Bq4fWsdQ1OCmBq7SREnLup/pnu17N0zEvqP30sRan1+C9Tj4rj22Ohy1tBBqQ0Fc Bn7AI634gAt0n05bM3u5RErkj1SUqFksxExcarFVHwVT929e2ljiqUngr8OHHYSk KaEO/3OhPjg= =J34N -----END PGP SIGNATURE-----
On 2007-01-03 00:35, Barry Warsaw wrote:
On Jan 2, 2007, at 5:41 PM, M.-A. Lemburg wrote:
Note that as side-effect of this it becomes a lot harder to manipulate PYTHONPATH to trick Python into loading a standard module from a non-standard location, improving security and robustness of the Python installations.
Sometimes though you want to do this, as when you want your application to ensure it gets a particular version of a standard library module, regardless of the version of Python being used. And now we're back to application-specific site-packages ;).
Well, I guess that's a rather particular use case and can probably only be safely implemented by the maintainer of the module or package in question ;-) In such (rare) cases, it should be possible to use one of the harder ways to achieve this: * monkey patching the package * using package.__path__ to redirect the in-package search * creating a private copy of the whole package which then has the modified modules and packages in place Regarding application specific package setups: In my experience it's better to have an application specific sys.path setup function that manages this, rather than trying to manipulate PYTHONPATH or trying to tweak Python's stdlib site.py into using some particular way of setting up application specific paths which then makes interop harder for all applications using Python, rather than just the few that require such setups. The application can then call this path setup function early on in the startup phase to make sure that the rest of the startup and the application's main code then imports the right modules and packages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 03 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 3, 2007, at 6:07 AM, M.-A. Lemburg wrote:
Regarding application specific package setups:
In my experience it's better to have an application specific sys.path setup function that manages this, rather than trying to manipulate PYTHONPATH or trying to tweak Python's stdlib site.py into using some particular way of setting up application specific paths which then makes interop harder for all applications using Python, rather than just the few that require such setups.
The application can then call this path setup function early on in the startup phase to make sure that the rest of the startup and the application's main code then imports the right modules and packages.
Oh, I totally agree MAL. It's what I've done in Mailman for ages. What makes it more complicated is when you have dozens of entry points (read: command line scripts), but it's solvable. I guess when I read "PYTHONPATH" I also read "sys.path", so now that I've slept a little, never mind! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRZur2HEjvBPtnXfVAQIZjwP/Zbcz/aUooJtca/apUSkHwfhnTvMOLiiQ uoWOltYJnwqy3S9EYpUoan0rXBVPd04ygWf9tgZiioTaVHAuXYTLL7SikpiQTxge VxLQA6AegHlGMFgtuqTKNYDNeG2B9dlpHbT05ZSVqVaBeWi6E/ap/NNk7ufZ//pD 3F94t07yDaE= =OHMX -----END PGP SIGNATURE-----
participants (4)
-
Barry Warsaw -
Brett Cannon -
M.-A. Lemburg -
Steve Holden