[Python-Dev] Python equivalents in stdlib Was: Include datetime.py in stdlib or not?

[Brett Cannon]

On Wed, Jul 7, 2010 at 08:29, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Tue, Jul 6, 2010 at 11:54 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 7/6/2010 3:59 PM, Alexander Belopolsky wrote:
>>
>> I am more interested in Brett's overall vision than this particular module.
>> I understand that to be one of a stdlib that is separate from CPython and is
>> indeed the standard Python library.
>>
>
> I am also very much interested in the overall vision, but I would like
> to keep the datetime.py thread focused, so I a going to reply to broad
> questions under a separate subject.
>
>> Questions:
>>
>> 1. Would the other distributions use a standard stdlib rather than current
>> individual versions?
>
> I certainly hope they will.  In the ideal word, passing test.regrtest
> with unmodified Lib should be *the* definition of what is called
> Python.  I understand that there is already some work underway in this
> direction such as marking implementation specific tests with
> appropriate decorators.
>
>>
>> 2. Would the other distributions pool their currently separate stdlib
>> efforts to help maintain one standard stdlib?
>
> I believe that making stdlib and test.regrtest more friendly to
> alternative implementations will go long way towards this goal.  It
> will, of course, be a decision that each project will have to make.
>
>>
>> 3. What version of Python would be allowed for use in the stdlib? I would
>> like the stdlib for 3.x to be able to use 3.x code. This would be only a
>> minor concern for CPython as long as 2.7 is maintained, but a major concern
>> for the other implementation currently 'stuck' in 2.x only. A good 3to2
>> would be needed.
>
> Availability of python equivalents will hopefully help  "other
> implementation currently 'stuck' in 2.x only" to get "unstuck" and
> move to 3.x.   I understand that this is a somewhat sensitive issue at
> the moment, but I believe a decision has been made supporting new
> features for 2.x is outside of python-dev focus.
>
>
>> 4. Does not ctypes make it possible to replace a method of a Python-coded
>> class with a faster C version, with something like
>>  try:
>>    connect to methods.dll
>>    check that function xyx exists
>>    replace Someclass.xyy with ctypes wrapper
>>  except: pass
>> For instance, the SequenceMatcher heuristic was added to speedup the
>> matching process that I believe is encapsulated in one O(n**2) or so
>> bottleneck method. I believe most everything else is O(n) bookkeeping.
>>
> The ctypes modules is very CPython centric as far as I know.   For the
> new modules, this may be a valid way to rapidly develop accelerated
> versions.   For modules that are already written in C, I don't see
> much benefit in replacing them with ctypes wrappers.
>
>
>> [.. datetime specific discussion skipped ..]
>> From scanning that and the posts here, it seems like a pep or other doc on
>> dual version modules would be a good idea. It should at least document how
>> to code the switch from python version to the x coded version and how to
>> test both, as discussed.
>>
> I am certainly not ready to write such PEP.   I may be in a better
> position to contribute to it after I gain more experience with
> test_datetime.py.   At the moment I have more questions than answers.
>
> For example, the established practice appears to be:
>
> modulename.py
>
> # Python code
>
> try:
>    from _modulename import *
> except:
>    pass
>
> This is supposed to generate a .pyc file with no python definitions in
> it if  _modulename is available.  The problem with datetime.py is that
> it have several helper methods like _ymd2ord() that will still stay in
> the module.  Should an "else:" clause be added to clean these up?
> should these methods become class or static methods as appropriate?
>
> The  established practice for testing is
>
> py_module = support.import_fresh_module('modulename', blocked=['_modulename'])
> c_module = support.import_fresh_module('modulename', fresh=['_modulename'])
>
> class TestDefnitions: # not a unittest.TestCase subclass
>       def test_foo(self):
>            self.module.foo(..)
>       ...
>
> class C_Test(TestDefnitions, unittest.TestCase):
>       module = c_module
>
> class Py_Test(TestDefnitions, unittest.TestCase):
>       module = py_module
>
>
> For datetime.py this approach presents several problems:
>
> 1. replacing datetime with self.module.datetime everywhere can get
> messy quickly.
> 2. There are test classes defined at the test_datetime module level
> that subclass from datetime classes.  The self.module is not available
> at the module level.  These should probably be moved to setUp()
> methods and attached to test case self.
> 3. If #2 is resolved by moving definitions inside functions, the
> classes will become unpickleable and pickle tests will break.  Some
> hackery involving injecting these classes into __main__ or module
> globals may be required.

So I have been thinking about this about how to possibly make this
standard test scaffolding a little cleaner. I think a class decorator
might do the trick. If you had all test methods take a module argument
you could pass in the module that should be used to test. Then you
simply rename test_* to _test_*, create test_*_(c|py), and then have
those methods call their _test_* equivalents with the proper module to
test. You could even make this generic by having the keyword arguments
to the decorator by what the test suffix is named.

The benefit of this is you don't have to define one base class and
then two subclasses; you define a single test class and simply add a
decorator. This addresses #1. As for #3, that I can't answer and might
simply require restructuring those specific pickle tests.

-Brett


>
> These challenges make datetime.py an interesting showcase for other
> modules, so rather than writing a PEP based on abstract ideas, I think
> it is better to get datetime.py integrated first and try to establish
> the best practices on the way.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>