Mailman 3 March 2008 - Python-Dev

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 20, 2008

March 20, 2008

At 09:33 AM 3/20/2008 +0000, Paul Moore wrote: >1. No integration with the system packager (Windows, in my case). If I >do easy_install nose, then nose does not show up in add/remove >programs. That significantly affects the way I manage my PC. The long-term fix here is probably to have a platform-specific installer capable of either turning eggs into .msi or .exe installers, or of doing the add/remove programs integration directly. (Someone, of course, will have to step up to create such a tool.) >5. Auto-discovery doesn't always work. I'm sorry, I really can't >recall the example at the moment, but sometimes easy_install says it >can't find a package I *know* is available. Sometimes it does that to me, too. But then I look at the project's page in PyPI, and they don't have a link to a download page. Usually, they've got a link to a page on their site with instructions about downloading, but that doesn't directly link to any tarballs or anything. So I grab the link of the real download page and paste it into a -f option to easy_install, so it knows where to find the link from.

1 0

Capsule Summary of Some Packaging/Deployment Technology Concerns
by Jeff Rush March 20, 2008

March 20, 2008

I was in a Packaging BoF yesterday and, although not very relevant to the packager bootstrap thread, Guido has asked me to post some of the concerns. The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu and such. Everyone had strong expressions of frustration with the status quo and most had tried to resolve their issues but had their patches rejected. I am not taking either side and whether those rejections were justified I cannot say, but the general feeling of their concerns intentionally not being addressed isn't healthy. Several had abandoned setuptools, deeming it a failed solution and others called for a fork. To start, I am not a leader of the group nor do I claim I accurately captured and expressed all their concerns. I apologize to those in the BoF for any misrepresentations. 1. Many felt the existing dependency resolver was not correct. They wanted a full tree traversal resulting in an intersection of all restrictions, instead of a first-acceptable-solution approach taking now, which can result in top-level dependencies not being enforced upon lower levels. The latter is faster however. One solution would be to make the resolver pluggable. 2. People want a solution for the handling of documentation. The distutils module has had commented out sections related to this for several years. 3. A more flexible internal handing of the different types of files is needed. Currently the code, data, lib, etc. files are aggregated at build time and people would like them to be kept separate until install/packaging time. They also want greater flexibility in the kinds of files identified for packaging. There is currently a single plugin entrypoint for file_finding, so people have resorted to abusing the setuptools function find_packages() again and again with different include/exclude args. A solution is to expand the set of entrypoints into finer grained categories. They also want a way to expand the set of categories rather than a fixed set, which can be easily done with entrypoint groups and names. People also want a greater variety of file_finders to be included with setuptools. Instead of just CVS and SVN, they want it to comprehend Mercurial, Bazaar, Git and so forth. 4. They want an uninstall setuptools command. Adding one to remove a specific egg isn't difficult but correctly removing those dependencies that came in with that egg, without breaking later installs can be tricky. This is complicated because there isn't a single global package namespace to manage, when you factor in virtualenv and buildout sandboxes and per-user package areas. This differs from how RPMs and .debs are viewed. 5. There was concern over the .pth mechanism used by setuptools re activation. First, there is a (perceived) performance issue with increasingly adding every ZIP file explicitly onto sys.path. This may or may not be a red herring. The other is the use of a single .pth file to control the list of activated packages. Those who produce distributions would prefer a magic directory into which links to distributions could be dropped, similar to the current best practices for Linux, with /etc/conf.d/, /etc/profile.d/, /etc/xinetd.d/ and so forth. 6. There is a need for more extensibility hooks. People want places to plug in special handling. For example: a) setuptools has a --record option to capture the list of files installed for use by subsequent packaging tools. Some want that list to be available to a setuptools plugin. b) some want hooks for post-build/post-install actions, instead of the current approach of writing a custom build class that handles it all. 7. Many wanted to ability to install files anywhere in the install tree and not just under the Python package. Under distutils this was possible but it was removed in setuptools for security reasons. Custom code can still be written to do this explicitly but this is not popular. Neither setuptools nor distutils has the ability to rename files at install time. A fair question is whether it is the job of setuptools (or any Python packaging solution) to cover all these bases. The risk of not doing the job is that some of those in attendance were rolling their own solutions which do not play well with packages installed using other means, not seeing them. Distutils has intentionally tried to -not- be a general replacement packaging solution, with its support of the "bdist" command for various platform-specific distribution formats. We should continue not trying to replace platform-specific packaging technologies but perhaps improve our control of their creation. As mentioned, some of these concerns can be resolved by adding customization-pressure-release entrypoints to setuptools, and some can be handled with much better documentation of use cases and what to do. And some of it is confusion over packaging libraries versus applications, where setuptools focuses on the former and zc.buildout focuses on the latter. But buildout is very young, maintains isolation from the system Python and was not known to many of the packaging BoF attendees. Some of this may seem down on eggs, but I think they are really cool and would like to see them adopted as the standard for packaging Python software. There are rough spots on setuptools and buildout that would benefit from opening up the process and bringing in more developers, and communicating what they are and more importantly, what they are not. I believe the lack of a coherent packaging and deployment story for Python is hurting its uptake in many sectors and would like to work with others to strengthen this area of Python. -Jeff

7 14

PyCon: please review miy pending patches
by Christian Heimes March 20, 2008

March 20, 2008

Dear sprinters! I've a batch of pending patches I like to get into Python before the next alphas are send out. The math, epoll/kqueue and shell folder patches just need a review. The memory management patches are more complex. Please refer to the thread "int/float freelists vs pymalloc", too. epoll and kqueue patch: http://bugs.python.org/issue1657 Mark's and my math branch: trunk$ svnmerge.py merge -S svn+ssh://pythondev@svn.python.org/python/branches/trunk-math Windows shell folder patch for os.path: http://bugs.python.org/issue1763 Memory management of ints, floats and longs http://bugs.python.org/issue2039 http://bugs.python.org/issue2013 I've also two pending PEPs. I like to see at least PEP 370 in Python 2.6 and 3.0. It's not as intrusive as PEP 370 and IMHO very useful for deployment of Python applications. Post import hooks http://www.python.org/dev/peps/pep-0369/ Per user site-packages directory http://www.python.org/dev/peps/pep-0370/ Christian

2 1

trunk buildbot status
by Trent Nelson March 20, 2008

March 20, 2008

Quick update on the status of the trunk buildbots: Failing: [x86 gentoo trunk (Neal Norwitz)] This has been failing at the same point for the past couple of days now: test_sqlite command timed out: 1800 seconds without output, killing pid 15168 process killed by signal 9 program finished with exit code -1 None of the other buildbots seem to be encountering the same problem. Neal, got any idea what's going on with this one? [alpha True64 5.1 trunk (Neal Norwitz)] test_tarfile started failing recently (within the past few days) with CRC checks. See http://www.python.org/dev/buildbot/trunk/alpha%20Tru64%205.1%20trunk/builds…. Greg updated the test such that it prints out some more detail about the failure so we're waiting on that at the moment. [hppa Ubuntu trunk (Matthias Klose)] This has been consistently failing in test_socketserver for as long as I can remember: test_socketserver make: *** [buildbottest] Alarm clock program finished with exit code 2 I just updated that test such that it waits 20 seconds instead of 3 seconds at the end of the test if the server hasn't shutdown -- waiting for the test results of this still. [x86 XP trunk (Joseph Armbruster)] This box didn't survive the recent build changes, but I can't figure out why, none of the other Windows boxes encounter this error: The following error has occurred during XML parsing: File: C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj Line: 179 Column: 1 Error Message: Illegal qualified name character. The file 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' has failed to load. Can someone check a clean trunk build on a Windows system that *only* has Visual C++ Express 2008? The latest build system updates don't rely on any features of Visual Studio Professional, but the tools use a lot of common files, and perhaps a Service Pack needs to be applied or something. [amd64 XP trunk (Thomas Heller)] Builds fine, all tests pass except for test_largefile, which is failing as there's no more space left on the drive ;-) [x86 XP-4 trunk (David Bolen)] This is currently flagged as having failed test, but I don't think it's finished building since the finalised build updates, so hopefully the BSDDB errors in the last run will be resolved when it finished the latest build. [x86 FreeBSD 2 trunk (Jeroen Ruigrok van der Werven)] This is a FreeBSD 6.3-STABLE box (which switched to curses 5.6 from 5.2) -- there's been an ongoing thread with regards to why curses has started failing, Jeroen can probably provide more info on that. Either way I don't anticipate a quick fix for this particular slave, unfortuantely. Neal/Martin, I'd like to promote the following slaves to the stable list: [g4 osx.4] [x86 W2k8] [AMD64 W2k8] [ppc Debian unstable] [sparc Ubuntu] [sparc Debian] [PPC64 Debian] [S-390 Debian] [x86 XP-3] [amd64 XP] [x86 FreeBSD] [x86 FreeBSD 3] The trunk builds of these slaves have been the most reliable since I've been tracking. Trent.

5 7

Pretty-printing 2to3 Nodes
by David Wolever March 20, 2008

March 20, 2008

Would anyone be averse to changing pytree.Node's __repr__ so it includes the name of the name of the symbol the node represents? The only downside is that it makes the __reprs__ longer... But I think its worth the length: Node(313:simple_stmt, [Node(298:import_name, [Leaf(1, 'import'), Node (279:dotted_as_name, [Node(281:dotted_name, [Leaf(1, 'foo'), Leaf(23, '.'), Leaf(1, 'bar')]), Leaf(1, 'as'), Leaf(1, 'bang')])]), Leaf(4, '\n')]) OR just names: Node(import_name, [Leaf(1, 'import'), Node(dotted_as_name, [Node (dotted_name, [Leaf(1, 'foo'), Leaf(23, '.'), Leaf(1, 'bar')]), Leaf (1, 'as'), Leaf(1, 'bang')])]) OR the original: Node(313, [Node(298, [Leaf(1, 'import'), Node(279, [Node(281, [Leaf (1, 'foo'), Leaf(23, '.'), Leaf(1, 'bar')]), Leaf(1, 'as'), Leaf(1, 'bang')])]), Leaf(4, '\n')])

3 2

First green Windows x64 buildbots!
by Trent Nelson March 20, 2008

March 20, 2008

We've just experienced our first 2.6 green x64 Windows builds on the build slaves! Well, almost green. Thomas's 'amd64 XP trunk' ran out of disk: 304 tests OK. 1 test failed: test_largefile ====================================================================== ERROR: test_seek (test.test_largefile.TestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot\trunk.heller-windows-amd64\build\lib\test\test_largefile.py", line 42, in test_seek f.flush() IOError: [Errno 28] No space left on device Sorry about that Thomas ;-) Trent.

4 3

how to build extensions for Windows?
by Bill Janssen March 20, 2008

March 20, 2008

I've set up a Parallels virtual machine on my Mac, and have succeeded in getting Windows XP running in it! And I've installed MinGW, as well. Now I'd like to learn how to build the SSL module from source on Windows for Python 2.5.2. Is there any documentation on the process of building an extension from scratch that's appropriate for someone who doesn't know much about Windows? I'm looking for step-by-step. What about this? http://www.mingw.org/MinGWiki/index.php/Python%20extensions Bill

3 4

Order of Fixers
by David Wolever March 19, 2008

March 19, 2008

At the moment, fixers are run in alphabetical order -- but this poses a problem, because some depend on others (for example, fix_print will need to be run _before_ fix_future, because fix_print looks for the 'from __future__ import ...' statement. I'm tempted to simply change fix_future to fix_zz_future... But that has some obvious drawbacks. Alternately, if future is the only dependent module, it might be marginally cleaner to simply special-case it in refactor.get_all_fix_names. So, any better suggestions?

3 3

logging shutdown (was: Re: [Python-checkins] r61431 - python/trunk/Doc/library/logging.rst)
by Jim Jewett March 19, 2008

March 19, 2008

I think (repeatedly) testing an app through IDLE is a reasonable use case. Would it be reasonable for shutdown to remove logging from sys.modules, so that a rerun has some chance of succeeding via its own import? -jJ On 3/16/08, vinay.sajip <python-checkins(a)python.org> wrote: > Author: vinay.sajip > Date: Sun Mar 16 22:35:58 2008 > New Revision: 61431 > > Modified: > python/trunk/Doc/library/logging.rst > Log: > Clarified documentation on use of shutdown(). > > Modified: python/trunk/Doc/library/logging.rst > ============================================================================== > --- python/trunk/Doc/library/logging.rst (original) > +++ python/trunk/Doc/library/logging.rst Sun Mar 16 22:35:58 2008 > @@ -732,7 +732,8 @@ > .. function:: shutdown() > > Informs the logging system to perform an orderly shutdown by flushing and > - closing all handlers. > + closing all handlers. This should be called at application exit and no > + further use of the logging system should be made after this call. > > > .. function:: setLoggerClass(klass) > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

2 2

Improved thread switching
by Stefan Ring March 19, 2008

March 19, 2008

The company I work for has over the last couple of years created an application server for use in most of our customer projects. It embeds Python and most project code is written in Python by now. It is quite resource-hungry (several GB of RAM, MySQL databases of 50-100GB). And of course it is multi-threaded and, at least originally, we hoped to make it utilize multiple processor cores. Which, as we all know, doesn't sit very well with Python. Our application runs heavy background calculations most of the time (in Python) and has to service multiple (few) GUI clients at the same time, also using Python. The problem was that a single background thread would increase the response time of the client threads by a factor of 10 or (usually) more. This led me to add a dirty hack to the Python core to make it switch threads more frequently. While this hack greatly improved response time for the GUI clients, it also slowed down the background threads quite a bit. top would often show significantly less CPU usage -- 80% instead of the more usual 100%. The problem with thread switching in Python is that the global semaphore used for the GIL is regularly released and immediately reacquired. Unfortunately, most of the time this leads to the very same thread winning the race on the semaphore again and thus more wait time for the other threads. This is where my dirty patch intervened and just did a nanosleep() for a short amount of time (I used 1000 nsecs). I have then created a better scheduling scheme and written a small test program that nicely mimics what Python does for some statistics. I call the scheduling algorithm the round-robin semaphore because threads can now run in a more or less round-robin fashion. Actually, it's just a semaphore with FIFO semantics. The implementation problem with the round-robin semaphore is the __thread variable I had to use because I did not want to change the signature of the Enter() and Leave() methods. For CPython, I have replaced this thread-local allocation with an additional field in the PyThreadState. Because of that, the patch for CPython I have already created is a bit more involved than the simple nanosleep() hack. Consequently, it's not very polished yet and not at all as portable as the rest of the Python core. I now show you the results from the test program which compares all three scheduling mechanisms -- standard python, my dirty hack and the new round-robin semaphore. I also show you the test program containing the three implementations nicely encapsulated. The program was run on a quad-core Xeon 1.86 GHz on Fedora 5 x86_64. The first three lines from the output (including the name of the algorithm) should be self-explanatory. The fourth and the fifth show a distribution of wait times for the individual threads. The ideal distribution would be everything on the number of threads (2 in this case) and zero everywhere else. As you can see, the round-robin semaphore is pretty close to that. Also, because of the high thread switching frequency, we could lower Python's checkinterval -- the jury is still out on the actual value, likely something between 1000 and 10000. I can post my Python patch if there is enough interest. Thanks for your attention. Synch: Python lock iteration count: 24443 thread switches: 10 1 2 3 4 5 6 7 8 9 10 -10 -50 -100 -1k more 24433 0 0 0 0 0 0 0 0 0 0 1 1 6 0 Synch: Dirty lock iteration count: 25390 thread switches: 991 1 2 3 4 5 6 7 8 9 10 -10 -50 -100 -1k more 24399 10 0 0 0 0 1 0 1 0 975 1 1 0 0 Synch: round-robin semaphore iteration count: 23023 thread switches: 22987 1 2 3 4 5 6 7 8 9 10 -10 -50 -100 -1k more 36 22984 0 0 0 0 0 0 0 0 1 0 0 0 0 // compile with g++ -g -O0 -pthread -Wall p.cpp #include <pthread.h> #include <semaphore.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <assert.h> ////////////////////////////// // posix stuff class TMutex { pthread_mutex_t mutex; static pthread_mutex_t initializer_normal; static pthread_mutex_t initializer_recursive; TMutex(const TMutex &); TMutex &operator=(const TMutex &); public: TMutex(bool recursive = true); ~TMutex() { pthread_mutex_destroy(&mutex); } void Lock() { pthread_mutex_lock(&mutex); } bool TryLock() { return pthread_mutex_trylock(&mutex) == 0;} void Unlock() { pthread_mutex_unlock(&mutex); } friend class TCondVar; }; class TCondVar { pthread_cond_t cond; static pthread_cond_t initializer; TCondVar(const TCondVar &); TCondVar &operator=(const TCondVar &); public: TCondVar(); ~TCondVar() { pthread_cond_destroy(&cond); } void Wait(TMutex *mutex) { pthread_cond_wait(&cond, &mutex->mutex); } void TimedWait(TMutex *mutex, const struct timespec *abstime) { pthread_cond_timedwait(&cond, &mutex->mutex, abstime); } void Signal() { pthread_cond_signal(&cond); } void Broadcast() { pthread_cond_broadcast(&cond); } }; class TSemaphore { sem_t sem; TSemaphore(const TSemaphore &); TSemaphore &operator=(const TSemaphore &); public: TSemaphore(unsigned value) { sem_init(&sem, 0, value); } ~TSemaphore() { sem_destroy(&sem); } void Wait(); bool TryWait() { return sem_trywait(&sem) == 0; } void Post() { sem_post(&sem); } int Value() { int val; sem_getvalue(&sem, &val); return val; } }; pthread_mutex_t TMutex::initializer_normal = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t TMutex::initializer_recursive = PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP; TMutex::TMutex(bool recursive) { mutex = recursive ? initializer_recursive : initializer_normal; } pthread_cond_t TCondVar::initializer = PTHREAD_COND_INITIALIZER; TCondVar::TCondVar() { cond = initializer; } void TSemaphore::Wait() { while (sem_wait(&sem) == -1) assert(errno == EINTR); } ////////////////////////////// // misc stuff void Sleep(unsigned long ms) { struct timespec t; t.tv_sec = ms / 1000; t.tv_nsec = (ms % 1000) * 1000000; do { } while (nanosleep(&t, &t) < 0 && errno == EINTR); } class ScopedLock { TMutex &mutex; public: ScopedLock(TMutex &mutex):mutex(mutex) {mutex.Lock();} ~ScopedLock() {mutex.Unlock();} }; ////////////////////////////// // statistics struct Stats { static const int N_SLOTS = 15; int the_counts[N_SLOTS]; Stats(); void count(int slot); void print(); static int categorize(int slot); }; Stats::Stats() { for (int i=0; i<N_SLOTS; i++) the_counts[i] = 0; } void Stats::print() { printf(" 1 2 3 4 5 6 7 8 9 10 -10 -50 -100 -1k more\n"); for (int i=0; i<N_SLOTS; i++) printf("%6d", the_counts[i]); printf("\n"); } void Stats::count(int slot) { assert(slot >= 0); if (slot < N_SLOTS) the_counts[slot]++; else the_counts[N_SLOTS-1]++; } int Stats::categorize(int slot) { if (slot < 10) return slot; if (slot < 50) return 10; if (slot < 100) return 11; if (slot < 1000) return 12; if (slot < 10000) return 13; if (slot < 100000) return 14; if (slot < 1000000) return 15; return 100000; } Stats stats; ////////////////////////////// class TStandardPython { TSemaphore the_sem; public: TStandardPython(): the_sem(1) { } void Enter() { the_sem.Wait(); } void Leave() { the_sem.Post(); } bool NeedsSwitch() const { return true; } static const char *name() { return "Python lock"; } }; class TMyDirtyLock { TSemaphore the_sem; bool waitflag; public: TMyDirtyLock(): the_sem(1) , waitflag(false) { } void Enter(); void Leave() { the_sem.Post(); } bool NeedsSwitch() const { return waitflag; } static const char *name() { return "Dirty lock"; } }; void TMyDirtyLock::Enter() { if (waitflag) { struct timespec t; t.tv_sec = 0; t.tv_nsec = 1000; nanosleep(&t, &t); } waitflag = true; the_sem.Wait(); waitflag = false; } struct TRoundRobinSemaphore { struct LinkStruct { LinkStruct *queue_next; TCondVar wait; LinkStruct(): queue_next(0) { } }; private: TMutex the_lock; LinkStruct *wait_queue, *wait_last; public: TRoundRobinSemaphore(): wait_queue(0) { } void Enter(); void Leave(); bool NeedsSwitch() const { return wait_queue; } static const char *name() { return "round-robin semaphore"; } }; __thread TRoundRobinSemaphore::LinkStruct *t_linkstruct = 0; void TRoundRobinSemaphore::Enter() { if (!t_linkstruct) t_linkstruct = new LinkStruct; LinkStruct *ls = t_linkstruct; ScopedLock l(the_lock); // nobody there? -> done if (!wait_queue) { wait_queue = ls; wait_last = ls; return; } assert(wait_queue != ls); assert(wait_last->queue_next == 0); wait_last->queue_next = ls; wait_last = ls; while (wait_queue != ls) ls->wait.Wait(&the_lock); } void TRoundRobinSemaphore::Leave() { ScopedLock l(the_lock); LinkStruct *ls = t_linkstruct; assert(wait_queue == ls); wait_queue = ls->queue_next; if (wait_queue) { ls->queue_next = 0; wait_queue->wait.Signal(); } } struct sync_stuff { typedef TRoundRobinSemaphore LockInUse; LockInUse the_lock; int go_count; TMutex go_lock; TCondVar go_wait, go_wait2; bool gogogo; bool quit_now; int it_count; int sw_count; int last_thread; sync_stuff(): go_count(0) , gogogo(false) , quit_now(false) , it_count(0) , sw_count(0) , last_thread(-1) { printf("Synch: %s\n", LockInUse::name()); } }; struct thread_struct { sync_stuff *ss; int num; }; static void *the_thread(void *startup) { thread_struct *ss2 = static_cast<thread_struct*>(startup); sync_stuff *ss = ss2->ss; { ScopedLock l(ss->go_lock); ss->go_count++; ss->go_wait2.Signal(); while (!ss->gogogo) ss->go_wait.Wait(&ss->go_lock); } Sleep(10); bool firstrun = true; bool reenter = false; int last_count = -1; for (;;) { if (firstrun || reenter) ss->the_lock.Enter(); if (firstrun) Sleep(1); // do some work for (int i=0; i<10000; i++) ; if (last_count != -1) { int sl = ss->it_count - last_count; stats.count(Stats::categorize(sl)); } if (ss->last_thread != ss2->num) ss->sw_count++; ss->last_thread = ss2->num; last_count = ++ss->it_count; reenter = ss->the_lock.NeedsSwitch(); if (reenter) ss->the_lock.Leave(); firstrun = false; if (ss->quit_now) { if (!reenter) ss->the_lock.Leave(); break; } } { ScopedLock l(ss->go_lock); ss->go_count++; ss->go_wait2.Signal(); } return 0; } int main() { sync_stuff stuff; int go_num; for (int i=0; i<2; i++) { pthread_t tid; thread_struct *ss = new thread_struct; ss->ss = &stuff; ss->num = i+1; pthread_create(&tid, NULL, the_thread, ss); go_num = i+1; } { ScopedLock l(stuff.go_lock); while (stuff.go_count != go_num) stuff.go_wait2.Wait(&stuff.go_lock); stuff.go_count = 0; stuff.gogogo = true; stuff.go_wait.Broadcast(); } Sleep(1000); { ScopedLock l(stuff.go_lock); stuff.quit_now = true; while (stuff.go_count != go_num) stuff.go_wait2.Wait(&stuff.go_lock); } printf("iteration count: %d\n", stuff.it_count); printf("thread switches: %d\n", stuff.sw_count); stats.print(); return 0; }

2 4