Mailman 3 March 2009 - Python-Dev

Roundup / Python Tracker enhancements
by Daniel (ajax) Diniz 24 Mar '09

24 Mar '09

This proposal has two main goals: making the Python bug tracker more efficient for core developers and improving Roundup in areas that don't directly concern the PSF trackers. Most of the code would land in Roundup's repositories, but many instance-level changes would be specific to our tracker. Justification Python Tracker Some feedback is necessary to find out unvoiced problems, shortcomings and RFEs core developers have in mind regarding the tracker. From my work on old issues and with a couple of meta-tracker tickets, I can highlight a few pain points. Quoting myself: [A] relevant time-sink consists of inadequacies of the current [tracker] interface. Many search features are hard to use or notice, among them date spans and entering multiple inputs as lists. Other search features are lacking, mostly simple boolean relations, e.g. those including more than one keyword, full search terms, type, component, etc. Besides searching, the lack of interfaces (and backend support) for selecting and working with multiple issues tends to waste considerable amounts of time. To that, I can add some RFEs and bugs with the email interface, support for third-party authentication and many possible UI improvements. See a list of these below. Roundup The project has suffered from lack of developer time for a while and has grown a list of important improvements that would help it's adoption and revitalize the community. Correct handling of different charsets, DB performance and alternative indexers and templating engines are examples of often requested features. Making Roundup a better tool for developers and end-users, with a productive community can only benefit the PSF. Pre-Requisites and Status We have been working in this area recently, so many ideas are well developed, with a few fleshed out in great detail. Regarding the Python tracker, Martin is guiding my work on simple tracker issues, Brett gave us documents and many discussions about the issue handling workflow, Tennessee Leeuwenburg and R. David Murray are willing to triage issues and contribute to a better workflow. On the Roundup side, Stefan Seefeld and Richard Jones are leading new wave of features and fixes, with contributions from Tobias Herp, Bernhard Reiter and others. Student With the caveat that I'm probably unable to assess this objectively, a student should be comfortable with Python and have a grasp of general bug/issue handling procedures. Familiarity with Roundup is a huge plus, but the code is newbie-friendly. Mentors Under the condition that enough attention is given to Roundup itself, Stefan Seefeld is willing to mentor this proposal. I'll be available to help mentors and the student in many tasks. Martin von Löwis would be able to discuss many important parts of the work. Deliverables: Patches for upstream Roundup and for the Python instance. For a given proposal, a major goal should be declared, e.g. 'improve and clean up the infrastructure' or 'work on integration with third party services and tools'. Under that goal, task groups like 'refactor the different frontends' or 'implement VCS integration' should organize the developement effort in somewhat isolated steps. Some suggested topics, with examples: * means the idea is well developed and suitable for implementation # means a simple task or group of tasks, easy to implement Clean up and improve the base infrastructure Audit and fix any potential security issues * Fix support of diverse character encodings Performance improvements Reduce the number of RDBMS calls Improve caching across requests Improve end-user UI and features Enhance search abilities, better indexing Boolean operators Make forms easier to use and understand Support upload of multiple files in one request *# Support for third-party authentication, e.g. OpenID * Add developer-focused features to Roundup Support for milestones, tasks, workgroups Integration with code review tools Integration with Rietveld * Update/implement [D]VCS integration SVN integration Mercurial integration Interface to continuous integration tools Buildbot Administration enhancements Mass-update/batch-editing support *# UI for Class administration Discussion of more detailed tasks and open issues on roundup-devel[5] is highly encouraged. [1] https://lists.sourceforge.net/lists/listinfo/roundup-devel

2 1

MutableSet and Set named methods
by Raymond Hettinger 24 Mar '09

24 Mar '09

Does anyone remember the reason that most of the named methods were omitted from the ABC for mutablesets and sets? The update() method in particular would be nice to have. RAymond

2 1

About adding a new iterator method called "shuffled"
by Roy Hyunjin Han 24 Mar '09

24 Mar '09

I know that Python has iterator methods called "sorted" and "reversed" and these are handy shortcuts. Why not add a new iterator method called "shuffled"? >>> for x in shuffled(range(5)): >>> print x >>> 3 >>> 1 >>> 2 >>> 0 >>> 4 Currently, a person has to do the following because random.shuffle() does not return the actual shuffled list. It is verbose. >>> import random >>> x = range(5) >>> random.shuffle(x) >>> for x in x: >>> print x

2 1

an unimportant question, ...
by Christian Tismer 23 Mar '09

23 Mar '09

... but I'm curious. Hi Guido, while working on Psyco, I stumbled over a log entry in modsupport.h: > > 19-Aug-2002 GvR 1012 Changes to string object struct for > interning changes, saving 3 bytes. > The change to stringobject was this (rev. 28308): Before: > typedef struct { > PyObject_VAR_HEAD > long ob_shash; > PyObject *ob_sinterned; > char ob_sval[1]; > } PyStringObject; After: > typedef struct { > PyObject_VAR_HEAD > long ob_shash; > int ob_sstate; > char ob_sval[1]; > } PyStringObject; Now, the internals are very clear to me. What I don't understand is where the three saved bytes should be. Thinking of the time where this change was made, I cannot imagine that this comment was about the size diff between pointer and int, and if this was meant, I still don't get how this could save three bytes? With unaligned ob_sval, structure packing and ob_sstate being unsigned char one could save 3 bytes, but we don't do that. Well, as said, this is no important question. I am just asking myself what I don't see here, or if the comment is just sub-optimal :-) all the best -- chris p.s.: won't make it to PyCon this time, see you soon at the piggies -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

6 7

Re: [Python-Dev] [Roundup-devel] PSF GSoC slot for Roundup?
by Stefan Seefeld 23 Mar '09

23 Mar '09

Daniel (ajax) Diniz wrote: > Hi, > I'm about to submit a proposal for the PSF to use one GSoC slot for > the Python bug tracker. I've collected a few items I think might be > interesting for Roundup itself, most of which are valid RFEs for our > tracker. Feel free to add more ideas to the list below. > I very much like the idea of Roundup participating in the GSoC. As I mentioned off-list, I'm likely to mentor another project on boost.org (specifically, boost.python). While I would certainly like to help, I'm not sure to what degree I would be able to. This may also depend on the proposal itself. Is there a place I need to sign up to be considered as a potential mentor ? > I'd like to know what people think should be prioritized should > Roundup get one or part of a GSoC slot. BTW, if anyone wants to submit > a separate proposal for Roundup and is willing to mentor it, posts to > python-dev are welcome. > There are many interesting points on your wishlist. In particular, I like a common theme that emerges from some of those: Enhancing Roundup to be useful as a development tool. I have started to work on a new tracker template, specifically targeted at development (with a schema including 'milestone', and 'task'). It might be interesting to consider how this could be enhanced, for example to integrate with versioning tools such as subversion (for which Richard already worked out bindings), CVS, GIT, etc. Or, integrate with automated build and test infrastructure (buildbot comes to mind). Another topic for improvement is performance / optimization: * Improve caching and instance sharing across multiple (web) requests * Fuse multiple SQL queries to reduce the number of RDBMS calls * ... I'm sure there are a great many things to be worked on. Thanks for proposing Roundup to be part in GSoC ! Stefan -- Stefan Seefeld CodeSourcery stefan(a)codesourcery.com (650) 331-3385 x718

1 0

Issue workflow doc is now live
by Brett Cannon 23 Mar '09

23 Mar '09

I sent this out as a Google Doc a while back, but I just did a proof-reading, converted it, and pushed it live to the python.org: http://www.python.org/dev/workflow/ . So now people who ever triage issues have a guide to follow if they are not sure how to set things. -Brett P.S.: Doing this doc has made me realize that our issue workflow still feels off and SF-influenced. If people want to talk at PyCon about it just let me know. I might try to get a discussion going during the language summit or at the sprints as I have some ideas on how to simplify thing but make it usable for more people (preview: I like how http://code.djangoproject.com/querydoes their tickets).

4 5

speeding up PyObject_GetItem
by Daniel Stutzbach 23 Mar '09

23 Mar '09

In Python 2.5, list objects were special-cased to skip PyObject_GetItem and go straight to PyList_GET_ITEM. That special case gave made other sequences 20% slower than lists for getitem operations. The special case was removed in Python 3 (haven't checked 2.6). Today I was tracing through how PyObject_GetItem works, trying to figure it why it's so much slower, to see if we can get back some of that speed without special-casing just one type. Below is a rough sketch of what PyObject_GetItem does for the common case of a sequence with a valid positive integer parameter, and my observations on where there is room for improvement. I'm willing to write patches and test them, but I wanted to get some general feedback first. After all, it may be the way it is for A Very Good Reason that I'm not aware of. ;-) The code paths for PyObject_SetItem and PyObject_DelItem are essentially the same, so any speed improvments to one could easily be applied to the other two operations. PyObject_GetItem ob i ob==NULL # can't be null anyway i==NULL # can't be null anyway is mapping? ob is sequence? ob PyIndex_Check i tp_as_number i tp_flags i tp_as_number->nb_index i PyNumber_AsSsize_t i PyNumber_Index i i==NULL # still can't be null PyLong_Check i i==NULL # still can't be null PyLong_Check i # redundant => v v == -1 PySequence_GetItem ob v ob==NULL # still can't be null is sequence? ob # redundant sq_item? ob sq_item ob v I think there are two avenues for speed improvement, both of which involve removing redundancy: 1) Assume the index is a PyLong until proven otherwise The PyIndex_Check in PyObject_GetItem looks pretty useless. If it returns false, PyObject_GetItem throws a type error. If we skipped the PyIndex_Check when it would have returned false, PyNumber_AsSsize_t would later call PyIndex_Check and throw a type error. Unless I'm missing something, this is a clear win and makes the code simpler. In PyNumber_AsSsize_t, we could speed up the common case by trying PyLong_Check first, and if it fails then calling PyNumber_Index. This change would make the common case faster, but make the code a few lines longer. 2) Remove some of the exessive checking for NULL unless Py_DEBUG is defined Many of the routines in abstract.c check their parameters for NULL, as a sanity check, and throw a SystemError if NULL is detected. I'm tempted to suggest making these checks only when Py_DEBUG is defined, but I suspect if you wanted it that way, you would have changed it already. ;) Assuming you really want the NULL checks in production Python, there are 5 checks for NULL even though there are only two parameters. Seems like overkill? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>

3 2

Multiprocessing on Solaris
by Christian Heimes 23 Mar '09

23 Mar '09

Hello fellow co-developers! Today I was in contact with a Python user who tried to compile pyprocessing - the ancestor of multiprocessing - on Solaris. It failed to run because Solaris is missing two features (HAVE_FD_TRANSFER and HAVE_SEM_TIMEDWAIT). Does anybody have a Solaris box at his disposal to test the settings? Neither Python 2.6 nor my backup have the correct settings for Solaris. Christian

5 16

Integrate BeautifulSoup into stdlib?
by Vaibhav Mallya 23 Mar '09

23 Mar '09

I haven't seen a lot of discussion on this - maybe I didn't search hard enough - but what are people's thoughts on including BeautifulSoup in stdlib? It's small, fast, and pretty widely-liked by the people who know about it. Someone mentioned that web scraping needs are infrequent. My argument is that people ask questions about them less because they feel they can just reinvent the wheel really easily using urllib and regexes. It seems like this is similar to the CSV problem from a while back actually, with everyone implementing their own parsers. We do have HTMLParser, but that doesn't handle malformed pages well, and just isn't as nice as BeautifulSoup. In a not-entirely-unrelated vein, has there been any discussion on just throwing all of Mechanize into stdlib? BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ mechanize: http://wwwsearch.sourceforge.net/mechanize/ Regards, Vaibhav Mallya

26 65

Py_ssize_t support for ctypes arrays and pointers
by Thomas Heller 23 Mar '09

23 Mar '09

I received some (so far unfinished) patches for ctypes that will allow to create arrays with more than 2**31 elements and that will eventually also support pointer offsets larger than int, on 64-bit platforms. Since I do not have a machine with so much memory: Does one of the buildbots allow to run tests for this feature, or do I have to wait for the snakebite farm? -- Thanks, Thomas

3 3