Mailman 3 August 2011 - Python-Dev

SWIG (was Re: Ctypes and the stdlib)
by David Beazley Aug. 29, 2011

Aug. 29, 2011

On Mon, Aug 29, 2011 at 12:27 PM, Guido van Rossum <guido(a)python.org> wrote: > I wonder if for > this particular purpose SWIG isn't the better match. (If SWIG weren't > universally hated, even by its original author. :-) Hate is probably a strong word, but as the author of Swig, let me chime in here ;-). I think there are probably some lessons to be learned from Swig. As Nick noted, Swig is best suited when you have control over both sides (C/C++ and Python) of whatever … [View More]code you're working with. In fact, the original motivation for Swig was to give application programmers (scientists in my case), a means for automatically generating the Python bindings to their code. However, there was one other important assumption--and that was the fact that all of your "real code" was going to be written in C/C++ and that the Python scripting interface was just an optional add-on (perhaps even just a throw-away thing). Keep in mind, Swig was first created in 1995 and at that time, the use of Python (or any similar language) was a pretty radical idea in the sciences. Moreover, there was a lot of legacy code that people just weren't going to abandon. Thus, I always viewed Swig as a kind of transitional vehicle for getting people to use Python who might otherwise not even consider it. Getting back to Nick's point though, to really use Swig effectively, it was always known that you might have to reorganize or refactor your C/C++ code to make it more Python friendly. However, due to the automatic wrapper generation, you didn't have to do it all at once. Basically your code could organically evolve and Swig would just keep up with whatever you were doing. In my projects, we'd usually just tuck Swig away in some Makefile somewhere and forget about it. One of the major complexities of Swig is the fact that it attempts to parse C/C++ header files. This very notion is actually a dangerous trap waiting for anyone who wants to wander into it. You might look at a header file and say, well how hard could it be to just grab a few definitions out of there? I'll just write a few regexs or come up with some simple hack for recognizing function definitions or something. Yes, you can do that, but you're immediately going to find that whatever approach you take starts to break down into horrible corner cases. Swig started out like this and quickly turned into a quagmire of esoteric bug reports. All sorts of problems with preprocessor macros, typedefs, missing headers, and other things. For awhile, I would get these bug reports that would go something like "I had this C++ class inside a namespace with an abstract method taking a typedef'd const reference to this smart pointer ..... and Swig broke." Hell, I can't even understand the bug report let alone know how to fix it. Almost all of these bugs were due to the fact that Swig started out as a hack and didn't really have any kind of solid conceptual foundation for how it should be put together. If you flash forward a bit, from about 2001-2004 there was a very serious push to fix these kinds of issues. Although it was not a complete rewrite of Swig, there were a huge number of changes to how it worked during this time. Swig grew a fully compatible C++ preprocessor that fully supported macros A complete C++ type system was implemented including support for namespaces, templates, and even such things as template partial specialization. Swig evolved into a multi-pass compiler that was doing all sorts of global analysis of the interface. Just to give you an idea, Swig would do things such as automatically detect/wrap C++ smart pointers. It could wrap overloaded C++ methods/function. Also, if you had a C++ class with virtual methods, it would only make one Python wrapper function and then reuse across all wrapped subclasses. Under the covers of all of this, the implementation basically evolved into a sophisticated macro preprocessor coupled with a pattern matching engine built on top of the C++ type system. For example, you could write patterns that matched specific C++ types (the much hated "typemap" feature) and you could write patterns that matched entire C++ declarations. This whole pattern matching approach had a huge power if you knew what you were doing. For example, I had a graduate student working on adding "contracts" to Swig--something that was being funded by a NSF grant. It was cool and mind boggling all at once. In hindsight however, I think the complexity of Swig has exceeded anyone's ability to fully understand it (including my own). For example, to even make sense of what's happening, you have to have a pretty solid grasp of the C/C++ type system (easier said than done). Couple that with all sorts of crazy pattern matching, low-level code fragments, and a ton of macro definitions, your head will literally explode if you try to figure out what's happening. So far as I know, recent versions of Swig have even combined all of this type-pattern matching with regular expressions. I can't even fathom it. Sadly, my involvement was Swig was an unfortunate casualty of my academic career biting the dust. By 2005, I was so burned out of working on it and so sick of what I was doing, I quite literally put all of my computer stuff aside to go play in a band for a few years. After a few years, I came back to programming (obviously), but not to keep working on the same stuff. In particularly, I will die quite happy if I never have to look at another line of C++ code ever again. No, I would much rather fling my toddlers, ride my bike, play piano, or do just about anything than ever do that again. Although I still subscribe the Swig mailing lists and watch what's happening, I'm not active with it at the moment. I've sometimes thought it might be interesting to create a Swig replacement purely in Python. When I work on the PLY project, this is often what I think about. In that project, I've actually built a number of the parsing tools that would be useful in creating such a thing. The only catch is that when I start thinking along these lines, I usually reach a point where I say "nah, I'll just write the whole application in Python." Anyways, this is probably way more than anyone wants to know about Swig. Getting back to the original topic of using it to make standard library modules, I just don't know. I think you probably could have some success with an automatic code generator of some kind. I'm just not sure it should take the Swig approach of parsing C++ headers. I think you could do better. Cheers, Dave P.S. By the way, if people want to know a lot more about Swig internals, they should check out the PyCon 2008 presentation I gave about it. http://www.dabeaz.com/SwigMaster/ [View Less]

5 4

Should we move to replace re with regex?
by Guido van Rossum Aug. 29, 2011

Aug. 29, 2011

I just made a pass of all the Unicode-related bugs filed by Tom Christiansen, and found that in several, the response was "this is fixed in the regex module [by Matthew Barnett]". I started replying that I thought that we should fix the bugs in the re module (i.e., really in _sre.c) but on second thought I wonder if maybe regex is mature enough to replace re in Python 3.3. It would mean that we won't fix any of these bugs in earlier Python versions, but I could live with that. However, I don't … [View More]

17 53

Re: [Python-Dev] peps: Add memory consumption table.
by Antoine Pitrou Aug. 28, 2011

Aug. 28, 2011

On Sun, 28 Aug 2011 20:13:11 +0200 martin.v.loewis <python-checkins(a)python.org> wrote: > > +Performance > +----------- > + > +Performance of this patch must be considered for both memory > +consumption and runtime efficiency. For memory consumption, the > +expectation is that applications that have many large strings will see > +a reduction in memory usage. For small strings, the effects depend on > +the pointer size of the system, and the size of the … [View More]

1 0

Should we move to replace re with regex?
by Guido van Rossum Aug. 28, 2011

Aug. 28, 2011

Someone asked me off-line what I wanted besides talk. Here's the list I came up with: You could try for instance volunteer to do a thorough code review of the regex code, trying to think of ways to break it (e.g. bad syntax or extreme use of nesting etc., or bad data). Or you could volunteer to maintain it in the future. Or you could try to port it to PEP 393. Or you could systematically go over the given list of differences between re and regex and decide whether they are likely to be … [View More]

1 0

Add from __experimental__ import bla [was: Should we move to replace re with regex?]
by Dj Gilcrease Aug. 27, 2011

Aug. 27, 2011

In the thread about replacing re with regex someone mentioned adding to __future__ which isnt a great idea as future APIs are already solidified, they just live there to give developer time to adapt their code. The idea of a __experimental__ area is good for any pep's or stliib additions that are somewhat controversial (API isnt agreed on, code may take a while to integrate properly, developer wants some time to hash out any edge case bugs or API clarifications that may come up in large scale … [View More]

3 2

Windows installers and %PATH%
by Andrew Pennebaker Aug. 26, 2011

Aug. 26, 2011

Please have the Windows installers add the Python installation directory to the PATH environment variable. Many newbies dive in without knowing that they must manually add C:\PythonXY to PATH. It's yak shaving, something perfectly automatable that should have been done by the installers way back in Python 1.0. Please also add PYTHONROOT\Scripts. It's where cool things like easy_install.exe are stored. More yak shaving. The only potential downside to this is upsetting users who manage … [View More]

5 7

Summary of Python tracker Issues
by Python tracker Aug. 26, 2011

Aug. 26, 2011

ACTIVITY SUMMARY (2011-08-19 - 2011-08-26) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2963 (+26) closed 21665 (+35) total 24628 (+61) Open issues with patches: 1288 Issues opened (44) ================== #12326: Linux 3: code should avoid using sys.platform == 'linux2' http://bugs.python.org/issue12326 reopened by georg.brandl #12788: … [View More]

1 0

DNS problem with ar.pycon.org
by Facundo Batista Aug. 25, 2011

Aug. 25, 2011

Sorry for the crossposting, but I don't know who admins the pycon.org site. it seems that something happened to "ar.pycon.org", it should point to the same IP than "pycon.python.org.ar" (190.228.30.157). Somebody knows who can fix it? BTW, how do I update that page? We're having the third PyCon in Argentina this year... Thank you! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

1 0

Re: [Python-Dev] [Python-checkins] devguide: #12792: document the "type" field of the tracker.
by Nick Coghlan Aug. 25, 2011

Aug. 25, 2011

On Tue, Aug 23, 2011 at 7:46 AM, ezio.melotti <python-checkins(a)python.org> wrote: > +security > + Issues that might have security implications. If you think the issue > + should not be made public, please report it to security(a)python.org instead. A link to http://www.python.org/news/security/ would be handy here, since that has the GPG key to send encrypted messages to the security list. Cheers, Nick. -- Nick Coghlan | &… [View More]

1 1

FileSystemError or FilesystemError?
by Antoine Pitrou Aug. 25, 2011

Aug. 25, 2011

Hello, When reviewing the PEP 3151 implementation (*), Ezio commented that "FileSystemError" looks a bit strange and that "FilesystemError" would be a better spelling. What is your opinion? (*) http://bugs.python.org/issue12555 Thank you Antoine.

19 18