On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 9:31 AM, Guido van Rossum wrote:
I would like Gregor Lingl's approval of turning turtle.py into a package. It might make some things harder for novices, e.g. trackebacks and just browsing the source code.
Also many people don't expect to find any code in a file named __init__.py (and most of the time I agree with this). But the alternative isn't so great either, assuming we'll want strict backwards compatibility (I wouldn't want the instructions in Gregor's or anyone's book to start failing because of this). You can't rename turtle to turtle/turtle.py either, because then there'd be horrible confusion between turtle/ and turtle.py.
IOW, yes, flat still seems better than nested here!
Thanks for saying this. It might be a good time to also have a discussion about what may be an emerging trend of wanting to split standard library modules into packages. I understand the urge to split longer modules into multiple source files but would like to mention some of the advantages of keeping modules in a single source file. Taking the decimal module as an example: * Right now, I can pull up the source and entire history from subversion by knowing just the module name: http://svn.python.org/view/python/branches/py3k/Lib/decimal.py?view=markup , and I can search the entire module with nothing more sophisticated than "find" in the web browser. That no longer works with unittest. * I can easily email the file to someone or copy it back to a previous version of Python because only one file is involved. To look at the code, I don't have to guess how someone would have partitioned it into decimal.context.utils.threadlocal.py or somesuch. I don't need a packaging tool to distribute a module (pyparsing and BeautifulSoup are two examples of tools that are easily managed by just being in one file). * Some editors are set up to easily dance across multiple files and directories as if they were a single unit; however, many editors are not. Most file viewers and editors work better with single files. Take two operations in IDLE for example. If I press Alt-M for open module, I can retrieve all of the source for decimal (but this won't work with the logging or unittest packages). Once the source is visible, I press Alt-C to bring up a class browser for quickly reviewing the source (but this also won't work with packages). Several Python editors have this capability (provided by the pyclbr module) but they are defeated by packaging. * The Lib directory is more easily grepped when it is flat. The more we nest the Lib directory, the harder it will be to search the sources (while excluding the test directory). Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. I just wanted to bring up some of the advantages of having a single file for a library module. It would not be a good thing if some of the newer committers embarked on taking modules over 2000 lines and split them into packages. Right now, we have hundreds of source files, but it wouldn't take long for someone on a packaging mission to turn that into thousands. If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. Raymond
On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote: ..
Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not.
This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant. What were the benefits that justified unittest.py split?
2010/10/26 Alexander Belopolsky <alexander.belopolsky@gmail.com>:
On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote: ..
Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not.
This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant. What were the benefits that justified unittest.py split?
Mostly it was huge. -- Regards, Benjamin
On Oct 26, 2010, at 12:18 PM, Benjamin Peterson wrote:
2010/10/26 Alexander Belopolsky <alexander.belopolsky@gmail.com>:
On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote: ..
Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not.
This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant. What were the benefits that justified unittest.py split?
Mostly it was huge.
Now, it's huge and split across multiple files so it's harder to search, the class browser won't work, and the full source cannot be brought up immediately using just the module name. The svn annotations and history are munged-up. The components were highly interdependent so now every file has to start with a set of cross-imports. Well, at least the __init__.py is not full of code. That's something. FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those. Raymond
On Tue, 26 Oct 2010 12:34:30 -0700 Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those.
I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules. Regards Antoine.
On Oct 26, 2010, at 09:54 PM, Antoine Pitrou wrote:
I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules.
And that seems perfectly reasonable to me, especially if the alternative were that Michael was so frustrated with a massive single .py file that it discouraged him from working on it. If done well, a split can help improve the readability and discoverability of the code. I shudder at the madness that a single email.py file would cause. -Barry
On Oct 26, 2010, at 8:37 PM, Barry Warsaw wrote:
If done well, a split can help improve the readability and discoverability of the code.
No doubt that is true. The problem is that splitting can also impair discoverability. When unittest was in one file, you knew the filename was unittest.py from just the module name. Now, there are ten files and you have no way to guess any of their names or how the author grouped the functions and methods inside them. Let's say that I'm mystified about the difference between assertSameElements() and assertItemsEqual() and assertSetEqual(). So, I go to http://svn.python.org/view/python/branches/py3k/Lib/unittest/ . Now, how do you know which file contains the source? Formerly, I could call-up the one source file and do a find. Or, I could do OpenModule: unittest with IDLE and instantly see the source and let the class browser analyze the structure of the file. Now, that's not possible either. The source for those methods is now less discoverable. In the unittest/case.py file, the safe_repr() function function is called over 40 times and it is not used in any other file. So how do be benefit from it being defined in the utils.py file? ISTM, this saved nothing. The case.py file is over 1000 lines long and utils is 80. How did we benefit from that split? For me, it makes it harder to read the code because I have to go looking elsewhere for commonly called functions. The unittest module grew from one file in Py2.6 to ten files and one directory with 2500 lines in Py2.7. Was that a win? I've spent time trying to read through the changes and cannot follow it without having most of those ten files open in my editor. For me, it's a PITA to read the code. It doesn't help that the file split blew away the svn blame history, so I have a harder time being about to tell what changed. All that being said, I started this thread with: "Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not". The goal of the post was just to raise awareness that splitting modules is not a straight win. There are costs to discoverability and searchability. It makes pyclbr less useful. It makes svn viewer less useful. And it loses history. The split of unittest is a done deal. Who knows, it may have been a net win. Am just hoping that we all understand that it was not cost free. My hope is that the splitting modules unnecessarily does not become a trend. Packages should be the last tool we reach for, not the first. In many cases, flat is better than nested. Raymond P.S. I liked your qualifier, "if done well". There's a lot in those three words.
On Tue, 26 Oct 2010 23:37:10 -0400, Barry Warsaw <barry@python.org> wrote:
On Oct 26, 2010, at 09:54 PM, Antoine Pitrou wrote:
I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules.
And that seems perfectly reasonable to me, especially if the alternative were that Michael was so frustrated with a massive single .py file that it discouraged him from working on it.
If done well, a split can help improve the readability and discoverability of the code. I shudder at the madness that a single email.py file would cause.
To put your mind at ease, Barry, I'd not want to do that either :) But by (IMO good) design Generator, FeedParser, and Message are all supposed to be independent (use only each other's public APIs). And Header can be (and is, I think) used without the other pieces of email, as is true for other of the helper modules (base64mime, quoprimime, etc). On the other hand, I have no clue why 'iterators.py' exists :) The one that bugs me most, though, is MIME. Combining all the mime stuff into one file seems like it would be a big win (not that it's possible, now). What is the benefit of email.mime.text.MIMEText over email.mime.MIMEText, when each of the files in the mime package consists of a single subclass? So, to clarify, like Raymond I'm not saying that packages are always bad. I'm just saying that packages are also not always good; and, further, that the number of lines of code in a file should, IMO, have nothing to do with the decision as to whether or not to create a package. -- R. David Murray www.bitdance.com
On Oct 27, 2010, at 10:34 AM, R. David Murray wrote:
To put your mind at ease, Barry, I'd not want to do that either :)
Phew! But I wasn't worried, 'cause I know you're not insane. (Though the fact that you've effectively inherited the email package does bring that into question. :)
But by (IMO good) design Generator, FeedParser, and Message are all supposed to be independent (use only each other's public APIs). And Header can be (and is, I think) used without the other pieces of email, as is true for other of the helper modules (base64mime, quoprimime, etc).
Agreed.
On the other hand, I have no clue why 'iterators.py' exists :)
They're there for some corner use cases, but they do tend to be helpful. body_line_iterator() is probably the one I've used the most. The iterators are intended to be used independently.
The one that bugs me most, though, is MIME. Combining all the mime stuff into one file seems like it would be a big win (not that it's possible, now). What is the benefit of email.mime.text.MIMEText over email.mime.MIMEText, when each of the files in the mime package consists of a single subclass?
I think you're right that the extra level of module path is probably unnecessary. I'm not sure that means all the .py files in email/mime should be combined though. OTOH, `wc -l Lib/email/mime/*.py` is only 314 lines so I'm happy to defer to you on that.
So, to clarify, like Raymond I'm not saying that packages are always bad. I'm just saying that packages are also not always good; and, further, that the number of lines of code in a file should, IMO, have nothing to do with the decision as to whether or not to create a package.
+1 -Barry
On 10/26/2010 02:34 PM, Raymond Hettinger wrote:
FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those.
Sense you mention this... I've worked on pydoc to make it much nicer to use in a browser. While doing that I needed to reworked the server part. That resulted in a clean server thread object (and supporting parts) with no pydoc specific code in those parts. It can work as a stand alone module quite nicely. It's about 170 lines with around a third of that as documented examples that can also run as doctests. More to the point, it's a simple text/html server wrapped in a thread object. It can work as a starting point to using a browser as a user interface like pydoc does. There is a patch in the bug tracker, I just need to make some minor updates to it and it can go in, but I really need some code organizing/placement review help. I I'm wonder what you may think. Keep it in pydoc or move it to the HTTP package? Document it or not? Ron
On Oct 26, 2010, at 2:54 PM, Ron Adam wrote:
I've worked on pydoc to make it much nicer to use in a browser.
While you're at it. Can you please modernize the html and create a style sheet? Right now, all of formatting is deeply intertwined with content generation. Fixing that would be a *huge* improvement. Raymond
On 10/26/2010 05:35 PM, Raymond Hettinger wrote:
On Oct 26, 2010, at 2:54 PM, Ron Adam wrote:
I've worked on pydoc to make it much nicer to use in a browser.
While you're at it. Can you please modernize the html and create a style sheet? Right now, all of formatting is deeply intertwined with content generation.
Fixing that would be a *huge* improvement.
Half way there! The server will read one if it exists. ;-) I'd really like to get this part in before 3.2 beta, and then I'll add a basic style sheet and update the html code to use it for 3.3. The present patch fixes and updates all the functional parts and allows you to do every thing that you can do on the command line, but a LOT easier. I think You, Nick. or one of the other Core developers could probably have this finished up in an afternoon if you really wanted. All the parts work. It's more about checking and adjusting the packaging at this point. Cheers, Ron
On 10/26/2010 05:35 PM, Raymond Hettinger wrote:
On Oct 26, 2010, at 2:54 PM, Ron Adam wrote:
I wonder what you may think. Keep it in pydoc or move it to the HTTP package? Document it or not?
I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server? Cheers, Ron
On Wed, Oct 27, 2010 at 2:33 PM, Ron Adam <rrr@ronadam.com> wrote:
I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server?
Given the time frame until beta 1, I'd suggest leaving it in pydoc for now. We can look at possibly documenting it and moving it to http.servers for 3.3. (The patch is attached to issue 2001 for anyone else that wants to take a look at it) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10/27/2010 08:51 AM, Nick Coghlan wrote:
On Wed, Oct 27, 2010 at 2:33 PM, Ron Adam<rrr@ronadam.com> wrote:
I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server?
Given the time frame until beta 1, I'd suggest leaving it in pydoc for now. We can look at possibly documenting it and moving it to http.servers for 3.3.
(The patch is attached to issue 2001 for anyone else that wants to take a look at it)
Cheers, Nick.
Fantastic! Thanks Nick
On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another.
+1 (or more) -- R. David Murray www.bitdance.com
On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettinger<raymond.hettinger@gmail.com> wrote:
If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. +1 (or more) Really, you prefer to work in single multi-thousand line modules than cleanly split smaller packages? (Is that how you develop your own
On 26/10/2010 15:05, R. David Murray wrote: projects, out of interest?) The cross imports are unfortunate (there are very few actually circular dependencies in the package though I think), but more of an indictment on the design of unittest than the decision to split it into a package. How the splitting happened is that I proposed it on this list many months ago and got one reply in favour and no dissenters. Benjamin actually did the splitting. I find unittest *massively* easier to maintain and navigate now that it is a package. It doesn't take very long to work out where the classes live (no time at all if you look in the module names - where the classes live is *very* obvious except for the TextTestResult which lives alongside the TestRunner). The package split into modules is into conceptual units, where the responsibility of the units is clear. This makes the code not only easier to navigate but easier to *think about*. The structure of unittest is now much clearer. I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. There is something to be said for consistency within the standard library, but I don't think it is a price worth paying in this particular case. All the best, Michael Foord
-- R. David Murray www.bitdance.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
On Tue, Oct 26, 2010 at 4:46 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers.
Gadzooks, Michael! Something else we agree on. 2000-line modules are a source of maintenance pain. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A storm broke loose in my mind." --Albert Einstein
On Tue, Oct 26, 2010 at 5:37 PM, Fred Drake <fdrake@acm.org> wrote:
On Tue, Oct 26, 2010 at 4:46 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers.
Gadzooks, Michael! Something else we agree on. 2000-line modules are a source of maintenance pain.
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files. If instead, unittest module was split into several top level modules so that test cases can be defined without pulling in test runner machinery and third party test runners could avoid importing irrelevant text runners, that would qualify as an improvement in my book, but probably not big enough to justify breaking compatibility. While maintainers' convenience is a valid valid concern and some level of idiosyncrasy is healthy to allow active maintainers to code in their preferred style, I think users' convenience should come first when it conflicts with that of maintainers. Remember, code is written once and read many. This is particularly true about stdlib. A minor inconvenience of finding the right place to stick a new function in a large file does not in my view overweights a major inconvenience of not having all pieces in one place neatly organized in a linear order.
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files.
Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. Regards Antoine.
On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files.
Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure.
I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises).
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon <brett@python.org> wrote:
So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises).
Something to note in PEP 8, perhaps? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Nov 2, 2010 at 15:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon <brett@python.org> wrote:
So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises).
Something to note in PEP 8, perhaps?
If everyone agrees with making this policy, then yes. -Brett
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 02, 2010, at 03:43 PM, Brett Cannon wrote:
On Tue, Nov 2, 2010 at 15:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon <brett@python.org> wrote:
So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises).
Something to note in PEP 8, perhaps?
If everyone agrees with making this policy, then yes.
If SHOULD not MUST, then +0 -Barry
On 02/11/2010 22:43, Brett Cannon wrote:
On Tue, Nov 2, 2010 at 15:33, Nick Coghlan<ncoghlan@gmail.com> wrote:
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon<brett@python.org> wrote:
So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes.
I'd like to reply a bit further, I'll do it as a reply to your earlier email though. Michael
-Brett
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Nov 2, 2010, at 3:33 PM, Nick Coghlan wrote:
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon <brett@python.org> wrote:
So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises).
Something to note in PEP 8, perhaps?
I'll propose some PEP 8 wording in the bug tracker (essentially advice on when and how to use packaging), and everyone can offer their assent, dissent, and word-smithing. Raymond
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote:
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is.
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Raymond
On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced.
This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package.
What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen?
To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? -- --Guido van Rossum (python.org/~guido)
On Nov 2, 2010, at 3:58 PM, Guido van Rossum wrote:
To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it?
I don't find anything offensive about it. The issues have to do with being able to find and analyze code. For example, to find-out what assert.ItemsEqual does, I have to figure-out that it was put in the case.py file. In Py2.6, you code use IDLE's Open Module tool to immediately bring up all the source for unittest. Then you could fire-up the class browser to quickly see and navigate the structure, but that also no longer works in Py2.7. Also, it used to be the just knowing the module name was sufficient to find the code with http://svn.python.org/view/python/branches/release26-maint/Lib/unittest.py?v... All you needed to study the code was a web browser and its find function. Now you need to open ten tabs to be able to browse this code. IOW, the packaging broke a read-the-source-luke style of research that I've been teaching people to use for years. I probably didn't articulate the above very well, but I think Martin said it more succinctly in this same thread. The other issue that Brett pointed out is that the file names now become part of the API, "from unittest.utils import safe_repr". In the logging module, packaging was done well. The files fell along natural lines in the API, some of the components we usable separately and testable separately. Likewise with the xml packages. In contrast, the unittest module is full of cross-imports and tightly coupled pieces (like suite and case) have been separated. Raymond
Le mardi 02 novembre 2010 à 16:20 -0700, Raymond Hettinger a écrit :
For example, to find-out what assert.ItemsEqual does, I have to figure-out that it was put in the case.py file.
Well, it's a TestCase method, so it seems rather intuitive to look for it in case.py. Regards Antoine.
On 02/11/2010 22:58, Guido van Rossum wrote:
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced.
This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package.
What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of
On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote: style for development myself. What is so offensive about it?
Amen. Although not that contrarian, others have spoken up in favour. The split is pretty obvious (in general - I'm sure its not perfect) and divided along major functionality. TestCase - case.py TestResult - result.py TestSuite - suite.py TextTestRunner - runner.py TestLoader - loader.py main function - main.py signal handling - signals.py The utils module is somewhat an odd one out as it is only used by case.py, but this is hardly the most egregious error in the world. If you can't guess where a class lives, __init__.py shows you explicitly (a clear advantage of exporting the public API at the top level ;-) Due to the original design of unittest (and I have many thoughts on that) the modules aren't strictly "reusable" (i.e. isolated from each other) - but many test frameworks (for example) just use the TestCase without using other components. I find it difficult to believe that this package structure is only acceptable if we make people import the TestCase from unittest.case and not expose it at the top level. As mentioned in another email, but this thread has many long and tedious emails, there is no clear consensus that there should be a remerger and I am aware that there are already some consumers of the new package structure. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process.
I don't think that anybody seriously expected the unittest package would be restructured again. The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. Personally I don't think it should -- it should by now be clear that this is not an area where one style will fit all. I also think that the convenience of one style over another might have something to do with the tools being used. -- --Guido van Rossum (python.org/~guido)
On Nov 2, 2010, at 4:43 PM, Guido van Rossum wrote:
The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules.
I was thinking of PEP 8 wording that listed the forces for and against. For example, ply.yacc and ply.lex was a very useful split (separately testable, natural division of concerns, no nested of cross-imports). The xml.sax, xml.dom, and xml.minidom was a nice split because it separated distinct tools. The xml packaging also worked well because it is easy to substitute in alternate parsers implementing the same API. I think we also want to recommend against putting much if any code in __init__.py. Some forces against packaging are that it breaks the class browser. As you say, different users of different toolsets are affected differently. For me, the unittest split broke my usual ways of finding out how the new methods were implemented. Another force against is what Brett pointed-out, that the package file structure becomes a permanent and unchangeable part of the API. It's a one-way street. In general, I think the advice should be that packaging should be done when there is some clear benefit beyond "turning one big file into lots of smaller files". Raymond
On Tue, Nov 2, 2010 at 5:03 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Some forces against packaging are that it breaks the class browser. As you say, different users of different toolsets are affected differently. For me, the unittest split broke my usual ways of finding out how the new methods were implemented.
Maybe the IDLE class browser can be fixed; there is plenty of code with this structure that can't or won't be restructured, no matter how strongly PEP 8 is worded. FWIW, personally I don't use the IDLE class browser -- I use Emacs, grep, and find. :-) -- --Guido van Rossum (python.org/~guido)
On Tue, Nov 2, 2010 at 16:43, Guido van Rossum <guido@python.org> wrote:
On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process.
I don't think that anybody seriously expected the unittest package would be restructured again. The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. Personally I don't think it should -- it should by now be clear that this is not an area where one style will fit all. I also think that the convenience of one style over another might have something to do with the tools being used.
This is not what I am suggesting for PEP 8. I want to say that a package's file structure should reflect the public API. I personally have no trouble with modules in packages that do not have a ton of objects in them. I just think if you have pkg/mod.py, pkg.mod should be exposed in the API, else name the file _mod.py. In the case of unittest that would just mean documenting that it's unittest.case.TestCase and that unittest.TestCase is for legacy reasons, much like os.path is just blindly added on to os even though it is a separate module(s).
On Tue, Nov 2, 2010 at 7:50 PM, Brett Cannon <brett@python.org> wrote:
This is not what I am suggesting for PEP 8. I want to say that a package's file structure should reflect the public API.
But what does that mean? I could argue that unittest's structure (TestCase in case.py, etc.) reflects its public API just fine.
I personally have no trouble with modules in packages that do not have a ton of objects in them. I just think if you have pkg/mod.py, pkg.mod should be exposed in the API, else name the file _mod.py. In the case of unittest that would just mean documenting that it's unittest.case.TestCase and that unittest.TestCase is for legacy reasons, much like os.path is just blindly added on to os even though it is a separate module(s).
I really don't think we should encourage the use as unittest.case.TestCase -- it's unnecessarily introducing structure. I think it's fine now that the cat is out of the bag to document unittest.case.TestCase as an alternative spelling, but I don't think it should be the preferred one. os.path is so old that should not be taken as an example for anything. (It predates packages!) But it should not be changed either, there'd be too much churn. -- --Guido van Rossum (python.org/~guido)
On Tue, Nov 2, 2010 at 6:58 PM, Guido van Rossum <guido@python.org> wrote: ..
To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it?
I would not call it "offensive", but what I find annoying is
import unittest unittest.TestCase.__module__ 'unittest.case'
This may not be a problem for smart tools, but for me and a simple editor what used to be: Let's find code for unittest.TestCase. 1. Open Lib/unittest.py. 2. Search for "class TestCase". is now 1. Open Lib/unittest.py -> No such file or directory. 2. OK, I'm in 2.7. Open Lib/unittest/__init__.py 3. Search for "class TestCase" -> beep 4. OK, search for "TestCase" -> from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf, .. 5. Hmm, what is ".case". Ah, the relative import - open case.py 7. Search for "class TestCase". 8. What exactly was I looking for? The above is only slightly exaggerated scenario that I went through several times when I started using 2.7 before I conditioned myself to grep in Lib/unittest/*.py. What is unfortunate is that file split was accompanied by an explosion of assert* methods in TestCase API which means that anyone reading 2.7 unittests is likely to encounter an unfamiliar method that has to be looked up. I think the problem that I described is just a slightly reworded problem that Raymond reported at the beginning of this thread. In other words, I am not alone in seeing this as a problem. PS: For a "made from scratch" API I would prefer TestCase only be available from unittest.case.
On Nov 3, 2010, at 11:26 AM, Alexander Belopolsky wrote:
This may not be a problem for smart tools, but for me and a simple editor what used to be:
Maybe this is the real problem? It's 2010, we should all be far enough beyond EDLIN that our editors can jump to the definition of a Python class. Even Vim can be convinced to do this (<http://rope.sourceforge.net/ropevim.html>). Could Python itself make this easier? Maybe ship with a command that says "hey, somewhere on sys.path, there is a class with <this name>. Please run '$EDITOR file +line' (or the current OS's equivalent) so I can look at the source code".
On Wed, Nov 3, 2010 at 4:59 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: ..
Maybe ship with a command that says "hey, somewhere on sys.path, there is a class with <this name>. Please run '$EDITOR file +line' (or the current OS's equivalent) so I can look at the source code".
Well, we already have inspect.findsource() for that.
Le mardi 02 novembre 2010 à 15:47 -0700, Raymond Hettinger a écrit :
What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen?
I don't think it's frozen. It's just that Michael and Benjamin (perhaps others too) prefer it like that, and given who does most of the maintenance and improvement work it is reasonable to respect that decision. If one day someone else becomes maintainer of unittest, then, sure, they can undo the splitting or do it differently. But right now there's no reason to change. Oh, and I much prefer a splitting without any impact on the public API. I *hate* writing "urllib.request.urlopen" and I really wish we hadn't done that; "urllib.urlopen" was so much easier to remember it isn't funny :/ Regards Antoine.
On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote:
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is.
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest".
Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase.
The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest".
I wouldn't say it was done badly, just non-standard. I was able to figure out where all the key classes were based on the file names. I personally would have no trouble doing ``from unittest.case import TestCase`` if more test case classes came along for various needs.
Also, I don't think the package structure was documented or announced.
Announced publicly? No, not that I know of.
This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing?
Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module.
Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen?
Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back.
On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote:
Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen?
Somewhat, yes.
That's a bummer. Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Raymond
Le mardi 02 novembre 2010 à 16:32 -0700, Raymond Hettinger a écrit :
On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote:
Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen?
Somewhat, yes.
That's a bummer.
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils).
I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
Antoine Pitrou <solipsis@pitrou.net> writes:
I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice.
If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)?
If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
I would say that names without a single leading underscore are part of the public API, whether documented or not. -- \ “Your [government] representative owes you, not his industry | `\ only, but his judgment; and he betrays, instead of serving you, | _o__) if he sacrifices it to your opinion.” —Edmund Burke, 1774 | Ben Finney
On 12:47 am, ben+python@benfinney.id.au wrote:
Antoine Pitrou <solipsis@pitrou.net> writes:
I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice.
If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)?
If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
I would say that names without a single leading underscore are part of the public API, whether documented or not.
And if that isn't the rule, then what the heck is? Jean-Paul
On Tue, Nov 2, 2010 at 8:47 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
I would say that names without a single leading underscore are part of the public API, whether documented or not.
I don't recall this ever being the standard library's policy. There are many modules using leading underscores as hints, and many others which don't. Other people consider the presence of a docstring on a non-underscored name significant, while still others refer to the out-of-line documentation as definitive. For modules, an __all__ attribute is generally agreed on as definitive, if present, but that's a fairly limited case. At this point, there isn't a single clear way to determine if something is public API. I doubt it will be likely to agree on a single definition now without engendering a lengthy discussion on whether names can be changed to reflect such a policy (where backward compatibility is sure to be lost). I'm sticking to the out-of-line documentation to determine what's public. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A storm broke loose in my mind." --Albert Einstein
On Wed, 03 Nov 2010 11:47:55 +1100 Ben Finney <ben+python@benfinney.id.au> wrote:
If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
I would say that names without a single leading underscore are part of the public API, whether documented or not.
That's not what we are talking about; we are talking about their locations. If the official location is the unittest package, then I don't see why we should also support undocumented locations just because they happen to work. Otherwise we should also support e.g. "unittest.unlink" if the unittest package happens to have "from os import unlink" at its top. I don't think it's reasonable. Antoine.
Antoine Pitrou <solipsis@pitrou.net> writes:
On Wed, 03 Nov 2010 11:47:55 +1100 Ben Finney <ben+python@benfinney.id.au> wrote:
If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
I would say that names without a single leading underscore are part of the public API, whether documented or not.
That's not what we are talking about; we are talking about their locations. If the official location is the unittest package, then I don't see why we should also support undocumented locations just because they happen to work.
So long as the names available for import are such that they indicate whether they're public or implementation-detail (i.e. without a leading single underscore or with one), I agree that this is distinct from the issue of locations on the filesystem.
Otherwise we should also support e.g. "unittest.unlink" if the unittest package happens to have "from os import unlink" at its top. I don't think it's reasonable.
Hmm. That example does give me pause. I'm trying to think of a simple way that such imports are excluded from being “public interface”, but can't immediately think of one. The distinction is clear in my head, though, for what it's worth :-) -- \ “I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.” —Douglas Adams | Ben Finney
Just a small input into this discussion: In EVE, for historical reasons, we implemented our own importing mechanism. I think it is because we started out with an ancient Python version that didn't support packages. Regardless, we still have a system where a hierarchy of files is scanned, and then code in each .py files determines where in the "namespace" it lands. This can be Declaratively (by using a __guid__ attribute on a class, for instance) or by defining a special __exports__ dict at the module level. The good thing about this system is that it allows us to separate code in a manner independent of the api. We can choose for example to group all network Code in a folder. Or have each class in the "game.entity" namespace be defined in its own file. It unhooks file structure from name structure. Now, this has its own problems of course, the biggest of it being that it is non-standard. Off the shelf IDEs have problems with it. And we have to implement dynamic reloading on our own. The list goes on, and for that reason, we are moving away from it in favor of standard python import. However, I am personally not super happy about how this will force one to think in "api" terms when creating source files. As has been mentioned, files cannot be moved and restructured once in general use, and when writing new code, one has to think long and hard about "where" to put the source, not "what" to put in it. What is more, a hierarchy, while a convenient system for storing files, does not, IMHO, always map to problem domain. But we're having a go at it. Time will tell if "forcing us to think inside the hierarchy" will be beneficial in the long run. Cheers, K
On 11/03/2010 01:47 AM, Ben Finney wrote:
If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
I would say that names without a single leading underscore are part of the public API, whether documented or not.
I understand this reasoning, but I'd like to offer counter-examples. For instance, would you say that glob.glob0 and glob.glob1 are public API? They're undocumented, they're not in __all__, but they don't have a leading underscore either, and source comments call them "helper functions." I'm sure there is a lot of other examples like that, both in the standard library and in python packages out there. Other than the existing practice, there is the matter of esthetics. Accepting underscore-less identifiers as automatically public leads to a proliferation of identifiers with leading underscores, which many people (myself included) plainly don't like.
On Nov 03, 2010, at 12:34 AM, Antoine Pitrou wrote:
I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff).
+1 -Barry
Raymond Hettinger <raymond.hettinger@gmail.com> writes:
Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? […] Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils).
Is this a case where it would be better if the package names had the leading underscore: ‘_utils’, ‘_suite’, etc.? Does the convention on single-leading-underscore identifiers as “don't rely on this name staying the same in future versions” hold for package names? -- \ “Alternative explanations are always welcome in science, if | `\ they are better and explain more. Alternative explanations that | _o__) explain nothing are not welcome.” —Victor J. Stenger, 2001-11-05 | Ben Finney
On Nov 03, 2010, at 11:06 AM, Ben Finney wrote:
Is this a case where it would be better if the package names had the leading underscore: ‘_utils’, ‘_suite’, etc.?
Does the convention on single-leading-underscore identifiers as “don't rely on this name staying the same in future versions” hold for package names?
I would vote "yes". I have seen more and more packages use this convention to signal that the module name is not intended to be imported directly. This should be part of any PEP 8 recommendation, IMO. -Barry
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils).
Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around).
Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... Michael
Cheers, Nick.
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
Le mercredi 03 novembre 2010 à 14:16 +0000, Michael Foord a écrit :
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around).
Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway...
unittest.TestCase is not really pickleable. There were test_multiprocessing issues because of that (see recent SVN checkins). Regards Antoine.
On 03/11/2010 14:17, Antoine Pitrou wrote:
Le mercredi 03 novembre 2010 à 14:16 +0000, Michael Foord a écrit :
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... unittest.TestCase is not really pickleable. There were test_multiprocessing issues because of that (see recent SVN checkins).
Interesting. We made some fixes before 2.7 to ensure they were copyable, but we fixed this in the copy module. TestCase instances now store some method objects in a dictionary which may make them unpickleable, so that could be a new problem. I'll test with 2.6 and 2.7 to see. An easy fix would be to store the method names rather than the method objects themself (if this is indeed the cause of the problem). This is what unittest2 does so that it works with earlier versions of Python that don't have the fix we put in copy. Michael
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
Le mercredi 03 novembre 2010 à 14:26 +0000, Michael Foord a écrit :
Interesting. We made some fixes before 2.7 to ensure they were copyable, but we fixed this in the copy module. TestCase instances now store some method objects in a dictionary which may make them unpickleable, so that could be a new problem. I'll test with 2.6 and 2.7 to see.
I don't think it is a problem in unittest, unless pickling TestCase objects is really useful. I have fixed the problem in test_multiprocessing instead. Regards Antoine.
On 03/11/2010 14:26, Michael Foord wrote:
On 03/11/2010 14:17, Antoine Pitrou wrote:
Le mercredi 03 novembre 2010 à 14:16 +0000, Michael Foord a écrit :
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... unittest.TestCase is not really pickleable. There were test_multiprocessing issues because of that (see recent SVN checkins).
Interesting. We made some fixes before 2.7 to ensure they were copyable, but we fixed this in the copy module. TestCase instances now store some method objects in a dictionary which may make them unpickleable, so that could be a new problem. I'll test with 2.6 and 2.7 to see.
An easy fix would be to store the method names rather than the method objects themself (if this is indeed the cause of the problem). This is what unittest2 does so that it works with earlier versions of Python that don't have the fix we put in copy.
from unittest import TestCase from pickle import dumps t = TestCase('assert_') dumps(t) "ccopy_reg\n_reconstructor\np0\n(cunittest\nTestCase\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'_testMethodDoc'\np6\nS'Fail
bigmac:beta.python.org michael$ python2.7 ActivePython 2.7.0.1 (ActiveState Software Inc.) based on Python 2.7 (r27:82500, Jul 4 2010, 13:58:56) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information.
from unittest import TestCase from pickle import dumps t = TestCase('assert_') dumps(t) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
Yep, looks like 2.7 introduced a bug making it impossible to pickle TestCase instances. I think it will be easy to fix, I'll create a specific issue: $python Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. the test unless the expression is true.'\np7\nsS'_testMethodName'\np8\nS'assert_'\np9\nsb." line 1374, in dumps Pickler(file, protocol).dump(obj) ... File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save rv = reduce(self.proto) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle instancemethod objects All the best, Michael
Michael
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around).
Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway...
Actually I think the answer is "no" (assuming you could pickle a TestCase). Here's an example with TestLoader: $ python27 Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25) [GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import unittest x = unittest.TestLoader() import pickle pickle.dumps(x) 'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
import pickle
$ python24 Python 2.4.4 (#1, Oct 23 2006, 13:58:00) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information. pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/pickle.py", line 1394, in loads return Unpickler(file).load() File "/usr/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/usr/lib/python2.4/pickle.py", line 1104, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.4/pickle.py", line 1138, in find_class __import__(module) ImportError: No module named loader The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib. -- Eric.
On 03/11/2010 14:53, Eric Smith wrote:
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around).
Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway...
Actually I think the answer is "no" (assuming you could pickle a TestCase). Here's an example with TestLoader:
Ah dammit, I read the question the other way round. Michael
$ python27 Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25) [GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import unittest x = unittest.TestLoader() import pickle pickle.dumps(x) 'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
import pickle
$ python24 Python 2.4.4 (#1, Oct 23 2006, 13:58:00) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information. pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/pickle.py", line 1394, in loads return Unpickler(file).load() File "/usr/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/usr/lib/python2.4/pickle.py", line 1104, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.4/pickle.py", line 1138, in find_class __import__(module) ImportError: No module named loader
The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib.
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 11/3/10 10:53 AM, Eric Smith wrote:
The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib.
And BTW: for me, this is the strongest reason not to break up modules into packages or otherwise reorganize the stdlib. -- Eric.
On Nov 3, 2010, at 11:25 AM, Eric Smith wrote:
On 11/3/10 10:53 AM, Eric Smith wrote:
The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib.
And BTW: for me, this is the strongest reason not to break up modules into packages or otherwise reorganize the stdlib.
This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) James
On Nov 3, 2010, at 1:04 PM, James Y Knight wrote:
This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :)
+1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts.
On 03/11/2010 14:53, Eric Smith wrote:
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around).
Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway...
Actually I think the answer is "no" (assuming you could pickle a TestCase). Here's an example with TestLoader:
It is actually fixable by temporarily switching the __module__ attribute of the classes inside a __reduce__ or __reduce_ex__ method. I couldn't see a cleaner way of doing it using the pickling protocol methods. I asked on #python-dev but the *only* person who claimed to understand the pickle protocol methods was Barry, and he is clearly insane. Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Although in practise this is less likely to be an issue for TestCase directly as it is extremely rare to use them without subclassing. More likely to be an issue for the test result or runner objects. All the best, Michael Foord
$ python27 Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25) [GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import unittest x = unittest.TestLoader() import pickle pickle.dumps(x) 'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
import pickle
$ python24 Python 2.4.4 (#1, Oct 23 2006, 13:58:00) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information. pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/pickle.py", line 1394, in loads return Unpickler(file).load() File "/usr/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/usr/lib/python2.4/pickle.py", line 1104, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.4/pickle.py", line 1138, in find_class __import__(module) ImportError: No module named loader
The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib.
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Wed, 03 Nov 2010 19:26:53 +0000 Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature...
Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine.
On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 03 Nov 2010 19:26:53 +0000 Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature...
Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are.
Regards
Antoine.
Splitting groups of tests to run in parallel via multiple processes is a pretty good use case.
Le mercredi 03 novembre 2010 à 15:48 -0400, Jesse Noller a écrit :
On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 03 Nov 2010 19:26:53 +0000 Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature...
Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are.
Regards
Antoine.
Splitting groups of tests to run in parallel via multiple processes is a pretty good use case.
Indeed, but it implies a lot of things about TestCase instances, which could have additional non-pickleable attributes (e.g. file objects). You'd better pickle the TestCase class instead, or simply the module name as we do with regrtest -jN. Regards Antoine.
On 03/11/2010 19:48, Jesse Noller wrote:
On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou<solipsis@pitrou.net> wrote:
Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of
On Wed, 03 Nov 2010 19:26:53 +0000 Michael Foord<fuzzyman@voidspace.org.uk> wrote: pickling a TestCase could be, and what the use cases are.
Regards
Antoine.
Splitting groups of tests to run in parallel via multiple processes is a pretty good use case.
That's something I've been thinking about a lot (and talking to Holger about) for the unittest plugins. I definitely won't be doing it with pickles but as Antoine says, sending test names to the subprocesses. You really want tests run in a child process to behave differently and it makes sense to set them up inside the child process. All the best, Michael Foord
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 02/11/2010 23:00, Brett Cannon wrote:
On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote:
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is.
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase.
It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.) Much better times to discuss this would be when it was proposed or when it was done, not months after it has been shipped in a production release.
[snip...]
This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module.
I *know* that some people are using the new package structure directly, because the topic has come up on the Testing in Python mailing list.
Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back. Absolutely, that would be the worst of all possible worlds - a monolithic huge module but still embedded in a package.
People *are* using the existing package structure to import directly from (against my advice!) as this particular topic has been discussed on the Testing In Python mailing list. Although Raymond has been vociferous in his detestation of this particular split he does not represent a "clear consensus" in favour of merging back. Both Fred Drake and Barrry Warsaw voiced their approval and on the "Clean up unittest API" issue both yourself (Brett) and Antoine have supported keeping the existing structure. Alexander Belopolsky and Martin Loewis expressed difficulties with the new structure, but that was in response to the original email from Raymond that didn't seem (on my reading) to expressly suggest re-merging unittest back into a module but was instead seemed to be using it as an example. I am aware of the costs of having a package rather than single (however large it may be) module, but to my mind the benefits to maintenance still outweigh these cost. I'm happy to again discuss these benefits at great length, but having had the same conversation in person with Raymond twice and at repeated most of the points (but by no means all) in this thread on the mailing list it really feels like going round in circles. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 02/11/2010 23:34, Michael Foord wrote:
On 02/11/2010 23:00, Brett Cannon wrote:
On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote:
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is.
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase.
It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.)
unittest2 dammit. Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
2010/11/2 Raymond Hettinger <raymond.hettinger@gmail.com>:
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote:
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is.
I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package.
See http://docs.python.org/whatsnew/2.7.html#updated-module-unittest -- Regards, Benjamin
On 02/11/2010 02:35, Brett Cannon wrote:
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky<alexander.belopolsky@gmail.com> wrote:
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to
On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou<solipsis@pitrou.net> wrote: this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began.
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't.
Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level.
He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files
Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward.
[snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API.
When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly. It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score. As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post. All the best, Michael Foord
I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Tue, Nov 2, 2010 at 19:50, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 02/11/2010 02:35, Brett Cannon wrote:
On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou<solipsis@pitrou.net> wrote:
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky<alexander.belopolsky@gmail.com> wrote:
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files.
Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure.
I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began.
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't.
Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split').
But not the public API as documented, e.g., it's documented as unittest.TestCase, not unittest.case.TestCase as the file structure suggests.
How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level.
I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -Brett
He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files
Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward.
[snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API.
When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly.
It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score.
As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post.
All the best,
Michael Foord
I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
--
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Tue, 2 Nov 2010 19:57:48 -0700 Brett Cannon <brett@python.org> wrote:
How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level.
I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail.
-1. unittest.TestCase is far simpler and more obvious that any javaesque qualified name. urllib.request and friends are already annoying enough. Regards Antoine.
Am 03.11.2010 03:35, schrieb Antoine Pitrou:
On Tue, 2 Nov 2010 19:57:48 -0700 Brett Cannon <brett@python.org> wrote:
How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level.
I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail.
-1. unittest.TestCase is far simpler and more obvious that any javaesque qualified name. urllib.request and friends are already annoying enough.
Agreed. There are about 30 names importable from unittest, that is quite manageable in a single namespace. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On 03/11/2010 02:57, Brett Cannon wrote:
On Tue, Nov 2, 2010 at 19:50, Michael Foord<fuzzyman@voidspace.org.uk> wrote:
On 02/11/2010 02:35, Brett Cannon wrote:
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky<alexander.belopolsky@gmail.com> wrote:
While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to
On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou<solipsis@pitrou.net> wrote: this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began.
I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). But not the public API as documented, e.g., it's documented as unittest.TestCase, not unittest.case.TestCase as the file structure suggests.
Right. I don't think that whether or not the unittest package structure is a good structure or not is determined by where we make users import the names from. Like others I see little value in reccommending people use the longer form for imports. All the best, Michael Foord
How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail.
-Brett
He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward.
[snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly.
It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score.
As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post.
All the best,
Michael Foord
I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
--
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
While maintainers' convenience is a valid valid concern and some level of idiosyncrasy is healthy to allow active maintainers to code in their preferred style, I think users' convenience should come first when it conflicts with that of maintainers. Remember, code is written once and read many. This is particularly true about stdlib. A minor inconvenience of finding the right place to stick a new function in a large file does not in my view overweights a major inconvenience of not having all pieces in one place neatly organized in a linear order.
I agree. While investigating an incompatibility in unittest2, I found that the breakage into multiple files makes it much harder to find out how things fit together, and where specifically a certain functionality is implemented. So join those who would have preferred this module to stay as a single 2000-line file. Regards, Martin
On Oct 26, 2010, at 04:46 PM, Michael Foord wrote:
I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. There is something to be said for consistency within the standard library, but I don't think it is a price worth paying in this particular case.
+1 (or more). :) or-maybe-you'd-prefer-more-^L-y y'rs, -Barry
On Tue, 26 Oct 2010 16:46:15 -0400, Michael Foord wrote:
On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettinger<raymond.hettinger@gmail.com> wrote:
If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. +1 (or more) Really, you prefer to work in single multi-thousand line modules than cleanly split smaller packages? (Is that how you develop your own
On 26/10/2010 15:05, R. David Murray wrote: projects, out of interest?)
Yes. Note that this is a more recent tendency, a few years ago I used to split things pretty finely right off the bat, but I found I wasted a lot of time on py-file interdependencies doing it that way. Now I generally only split when the pieces end up generalized enough that I want to reuse them. Which, granted, is often enough. But what I'm doing in that case is moving code out of a file at the top level of my library....into a new file at the top level of my library. Now, it's true that I haven't written any huge applications lately, but looking at a couple of typical projects the main files run between 1500 and 2000 lines (including comments), and I don't find those to be particularly big and certainly not awkward. I do understand the attraction of putting "related stuff" into separate files. I just think the balance may have tilted too far in the direction of nested and could stand to be nudged back toward flat :) Unittest is only 2600 lines total including comments. Typical file sizes are between 200 and 300 lines. Those feel tiny to me :) (And email is worse...a number of files only have a single class in them, and some of those classes consist of maybe 20 lines...) -- R. David Murray www.bitdance.com
participants (22)
-
"Martin v. Löwis" -
Alexander Belopolsky -
Antoine Pitrou -
Barry Warsaw -
Ben Finney -
Benjamin Peterson -
Brett Cannon -
Eric Smith -
exarkun＠twistedmatrix.com -
Fred Drake -
Georg Brandl -
Glyph Lefkowitz -
Guido van Rossum -
Hrvoje Niksic -
James Y Knight -
Jesse Noller -
Kristján Valur Jónsson -
Michael Foord -
Nick Coghlan -
R. David Murray -
Raymond Hettinger -
Ron Adam