Mailman 3 July 2009 - Python-Dev

Search error on tracker?
by Terry Reedy July 10, 2009

July 10, 2009

Before posting http://bugs.python.org/issue6453 I searched the tracker for 'bool TypeError' to avoid making a duplicate report. A few irrelevant hits were return but not the important one http://bugs.python.org/issue6428 which was the one I needed to find. Going back and searching titles for 'TypeError', cut and pasted from the title of my report, I get 6 hits but neither of the two above. Is there some sort of delay in updating the search indexes? Terry Jan Reedy

2 2

Summary of Python tracker Issues
by Python tracker July 10, 2009

July 10, 2009

ACTIVITY SUMMARY (07/03/09 - 07/10/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2245 open (+32) / 16035 closed (+17) / 18280 total (+49) Open issues with patches: 894 Average duration of open issues: 660 days. Median duration of open issues: 414 days. Open Issues Breakdown open 2215 (+32) pending 30 ( +0) Issues Created Or Reopened (52) … [View More]

1 0

"Absolute" paths in PEP 376 RECORD files
by P.J. Eby July 10, 2009

July 10, 2009

Just an idea... suppose that instead of using "real" absolute paths in the RECORD file for non-local files (scripts, data, etc) we changed the format to include a "prefix" field, containing something like LIBDIR, SHARE, SCRIPTS, etc., ala bdist_wininst internals? Also, we could include a separate (optional) PREFIXES file defining what those locations translated to at install time. Dumb bdists would still have their build paths there (or delete the file before packaging, or use real … [View More]

8 12

Re: [Python-Dev] PEP 376 - get_egginfo_files
by P.J. Eby July 9, 2009

July 9, 2009

At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >def get_distribution(name): > for d in get_distributions(): > if d.name == name: > return d > return None Btw, this is broken code anyway, because it's not handling case-insensitivity or name canonicalization. (I've mentioned these issue previously on the distutils-sig.)

4 3

Current patch process question
by Scott David Daniels July 8, 2009

July 8, 2009

I am confused by the current hydra-like development. I've recently submitted a change (issue6422 -- a feature request) which is currently moving through the process. I am not particularly asking for special attention on that issue, but I am confused about the process I should be using (in the face of two trunks). There seem to be two places to plausibly introduce a change: py3k and trunk. Should I submit changes for trunk, py3k, or both? I tried asking in the initial feature request, but … [View More]

3 3

Re: [Python-Dev] PEP 376 - Open questions
by P.J. Eby July 8, 2009

July 8, 2009

At 11:30 PM 7/7/2009 +0200, Tarek Ziadé wrote: >On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje(a)telecommunity.com> wrote: > > At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: > >> > >> When I started to work on this I didn't realize the gigantic amount of > >> work and coordination it requires > > > > No one expects the package inquisition. ;-) > > > > > >Sorry, i've looked in the english dictionary but I don't get this one. … [View More]

7 13

Re: [Python-Dev] PEP 376 - Open questions
by Paul Moore July 8, 2009

July 8, 2009

2009/7/8 Eric Smith <eric(a)trueblade.com>: > [Any reason you didn't respond to the list? I'd like to continue the > discussion there. But I'm currently replying just to you.] No, I just clicked the wrong button. Sorry. Redirecting back to the list. > Paul Moore wrote: >> >> 2009/7/8 Eric Smith <eric(a)trueblade.com>: >>> >>> I agree with this. For RPM's, there's a whole other database of what >>> files >>> were installed. … [View More]Is it really the intent that a file will be managed by >>> multiple different installers? That I can install with RPM but remove >>> with >>> some python-installer (or other) tool? That way lies madness. In fact, I >>> see >>> RECORD as an installer-specific detail that doesn't need to be >>> standardized >>> at all. >> >> I thought RECORD was to ensure that you can install with RPM, and >> *not* have any other random uninstaller remove your files (because you >> populate RECORD, and the other uninstaller uses >> pkgutil.get_file_users(path) to find out that you care about the file, >> and hence avoid deleting it). > > I really don't think this is a use case worth supporting. A single file > managed by multiple installers is a mess waiting to happen. RPM won't let > you do it. About the only use case I can think of is namespace packages, and > that can be handled by factoring out the __init__.py file into it's own > install package. Personally, I think you're right. But I have essentially zero experience with the non-Windows use cases, so I don't feel qualified to judge. It seems to me that a lot of the fancy detecting what other people do stuff is motivated by setuptools. I'm not going to comment on that, other than to say that it may be that setuptools is generating more problems than it's solving... >> Basically, other install methods can't read your (RPM) database, and >> you can't read their database. RECORD is a simple common format, >> providing just enough information for everyone to communicate the >> message "I manage this file". > > The RPM uninstaller won't look at the RECORD file. I think the best we can > say is that _if_ an installer wants to share state about what's installed > with other installers, this is how it's done. OK. I'd be stunned if bdist_wininst or bdist_msi ends up checking RECORD. So that's 3 "not interested" votes. Who's going to use the APIs based around the RECORD file? Anyone? Seriously - who is actually going to USE any of the APIs being proposed in PEP 376? If no-one's interested, we should stop right now. Do any of the people from the summit read this list? If not, can anyone contact them and get some feedback? Paul. [View Less]

1 0

Suggested PySys_SetArgv use with a (char **) argv ?
by Lisandro Dalcin July 8, 2009

July 8, 2009

Sorry... ---------- Forwarded message ---------- From: Lisandro Dalcin <dalcinl(a)gmail.com> Date: Tue, Jul 7, 2009 at 7:16 PM Subject: Re: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? To: "Martin v. Löwis" <martin(a)v.loewis.de> On Tue, Jul 7, 2009 at 6:46 PM, "Martin v. Löwis"<martin(a)v.loewis.de> wrote: >> Since many C applications take argv as a (char **) it seems reasonable >> for python to keep a function for the C api that accepts a &… [View More]

3 2

Re: [Python-Dev] PEP 376 - Open questions
by P.J. Eby July 8, 2009

July 8, 2009

At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: >When I started to work on this I didn't realize the gigantic amount of >work and coordination it requires No one expects the package inquisition. ;-)

5 4

Suggested PySys_SetArgv use with a (char **) argv ?
by Campbell Barton July 7, 2009

July 7, 2009

In Python3 PySys_SetArgv takes (wchar_t **) for the argv, I looked into converting a (char **) into a (wchar_t **) and while its possible its lengthy enough not to be trivial, see python.c:18 - char2wcharm(), its 102 lines with ifdef's and goto's, not including the loop lower down that loops over the argv. Looking further python has a very similar function for ./PC/bdist_wininst/install.c, for do_run_installscript - widen_string(). Since many C applications take argv as a (char **) it seems … [View More]

2 1

PEP 376 - RECORD format proposal
by Tarek Ziadé July 7, 2009

July 7, 2009

Hello, Let's state that we don't allow absolute paths in RECORD file and see what we can do with other paths. First as a reminder, here's the full list of directories used by Distutils at installation time. The "install" command uses five options to decide where to install the files. The default values are: Unix: install_purelib: sys.prefix/lib/python2.6/site-packages install_platlib: sys.exec_prefix/lib/python2.6/site-packages install_headers: sys.prefix/include/python2.6/docutils … [View More]

1 0

PEP 376 - Open questions
by Paul Moore July 7, 2009

July 7, 2009

As promised, here are some open questions on PEP 376. - Will the public API names be changed from *egginfo* to *metadata*? - What precisely are the use cases for absolute path names? Concrete examples are needed. With the current spec, some things can go wrong (e.g., see below), so we need real use cases to know how to address this. - How will bdist_wininst/bdist_msi/bdist_rpm be updated? - How will the RECORD file be managed? (Particularly for the case of bdist_xxx) [1] - Can distutils be … [View More]

7 15

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 7, 2009

July 7, 2009

At 08:43 PM 7/5/2009 +0200, Tarek Ziadé wrote: >But if it's based on PEP 302 protocols and if the pkgutil code works >with the sys.meta_path hook, >setuptools could then provide its loader, based on its EggFormats and >act as a provider without being broken. You misunderstand me. The whole point of putting .egg-info in distutils in the first place was to enable setuptools to detect the presence of disutils-installed packages. That's what's broken by changing the name.

6 17

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 7, 2009

July 7, 2009

At 03:13 PM 7/5/2009 +0200, Tarek Ziadé wrote: >The loader would still work with "DIST_NAME.egg-info" directories for >compatibility with >existing format in the query APIs, but the Distutils install command >would rather create "DIST_NAME.metadata" Note that this would then break setuptools without adding any benefit; ".metadata" is less precise and less unique than '.egg-info'. If you want a clearer name, '.pydist' or some such would at least be reasonably specific. (It'd … [View More]

6 10

Re: [Python-Dev] PEP 376 - Open questions
by P.J. Eby July 7, 2009

July 7, 2009

At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: >As promised, here are some open questions on PEP 376. > >- Will the public API names be changed from *egginfo* to *metadata*? +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff as.) >- What precisely are the use cases for absolute path names? Concrete >examples are needed. With the current spec, some things can go wrong >(e.g., see below), so we need real use cases to know how to address >this. The … [View More]purpose is to be able to remove scripts, data files, documentation, etc. that's stored in some place that's not package-relative, and survive a move of the package directory itself. For example, suppose I install with --prefix=~, and have files in ~/bin and ~/lib. I then move ~/lib/python2.7 to ~/pylib. If the scripts were installed with a relative path, this would break the link to the script still living in ~/bin. Now that I'm thinking about this, it occurs to me that there is another way to address this besides mixing relative and absolute paths in RECORD, and that is to make *all* the paths relative, and include an original installation location in INSTALLER. Then, it would be possible to detect whether the installation location had been moved and handle it accordingly... for some definition of "accordingly". I agree that further discussion is probably needed to resolve this. >- How will bdist_wininst/bdist_msi/bdist_rpm be updated? bdist_wininst, bdist_dumb, and various others use 'install --root' pattern to generate files for installation, which means that they would transparently end up writing a correct RECORD file, except for the inclusion of incorrect absolute paths for non-libdir-relative files. However, if we used the "all relative in RECORD, with a base in INSTALLER", these cases could transparently be treated as another instance of install directory relocation. I don't know if bdist_msi does a --root install before generating the .msi; if it does, then it should work the same way. >- Can distutils be made to install files in places the current RECORD >file spec can't handle? (I think the answer is "yes"). What happens >then? Surely an absolute path is sufficient? >- Should distribution names be case insensitive on case insensitive >filesystems? For comparison, module/package names are always case >sensitive even on case insensitive systems. IMO, they should be case-insensitive *regardless* of filesystem case, and I believe PyPI has already begun disallowing packages with only case-distinct names (e.g. 'Zope' vs. 'ZoPe') to be registered. [View Less]

3 2

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 7, 2009

July 7, 2009

At 07:15 PM 7/6/2009 +0100, Paul Moore wrote: >My point was that distributions which use setuptools in their >setup.py, *even if there's no runtime dependency on setuptools* end up >with non-standard .egg-info's. There is no good reason for this, from >my POV as a package user. So if setuptools is breaking otherwise >entirely standard packages, so that they won't work with core >distutils mechanisms, that's wrong. Setuptools is refusing to play >nicely with other packaging … [View More]systems. > >Now at the moment, this is a moot point, because the stdlib doesn't >have any package discovery mechanisms. Not so; it has .egg-info. (See footnote below.) >The PEP 302 hooks I'm writing provide the "plugin" ability for >alternative storage formats. But there's no provision for alternative >*metadata* formats. What's the point of a standard if people don't >follow it? I'm confused. Are you saying pkgutil will recognize .egg-info, or not? The only practical difference between the existing formats and the proposed new one (AFAICR) is the extension of the directory name, and the presence or absence of the RECORD file. My POV is that it would be a shame to discard compatibility with existing installed software, just for a bikeshedding of the directory name. Especially if it means that any tools that currently generate .egg-info's (and I'm not 100% positive setuptools is the only case) will need to grow options for whether to generate the old name, possibly based on what Python version... except you can install new distutils into old Python... etc. If this were strictly a matter of, "from Python 3.x on, generate .pydist", it's a lot less of a problem for me and everybody else, because Python 3 is effectively going to force a fork (or at least a branch) of setuptools anyway. For Python 2.x, having *two* names for .egg-info (which is what I understand the proposal to effectively be) is just asking for trouble, IMO. And my concern is that currently these compatibility issues are being handwaved rather than considered up front. To reiterate, summarize, and attempt to make as concrete as I can: I'm fine with a rename, *if and only if* it can be done in a way that never requires a tool to generate BOTH files for compatibility reasons. AFAICT, this can only be done by making the extension dependent upon the Python version, and this version distinction must be made *even if* the distutils themselves are backported to an earlier Python version. (Alternatively, this could be resolved by *not* backporting the PEP 376-compatible distutils/pkgutil to 2.x.) Currently, there is no official Python 3 release of setuptools, so compatibility can (IMO) be considered moot there. What's more, it is easy to conditionally determine whether to use '.egg-info' or '.pydist' based on Python version. What is not easy and likely to be confusing for everyone, is having the possibility for *both* to exist in the same system, *without* also having bidirectional interoperability. So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x and always .egg-info in 2.x", whether the latter part is achieved by making distutils/pkgutil use a version-dependent extension, or by refusing to backport distutils/pkgutil to 2.x. I'm -1 for having .pydist ever appear on a 2.x Python, either via stdlib or backporting. (These comments also apply for any name other than '.pydist'.) Footnote: there has been some discussion that implies that somehow .egg-info was hijacked into the stdlib, and that it should therefore not be supported. This is incorrect: at Pycon the year this happened (2005 IIRC), Guido *blessed setuptools for inclusion in the stdlib* in his keynote. Yes, *all* of setuptools, including easy_install, eggs, the whole nine yards. A few months later, when I actually checked in the code, there were protests from a few people on Python-Dev, some of which I agreed were valid. I reverted the checkin, floating instead a compromise proposal to add .egg-info support to distutils (i.e. the install_egg_info command) to allow setuptools and other 3rd party tools to detect the presence of distutils-installed packages. At the time, the assumption was that eggs and setuptools would still, eventually, make it into the stdlib -- at the time, Guido had not yet changed his mind about that. So the ".egg-info" name was not controversial, to the best of my recollection, and the inclusion did not violate any of the expressed concerns of the three persons who objected to including setuptools whole-hog. So, from my POV at least, dropping .egg-info is dropping backward compatibility on a feature that was *intentionally* added to the stdlib -- with approval from Guido and non-protest from Python-Dev -- specifically to provide that compatibility. That doesn't obligate it to stay... but I suggest that it *does* mean Python 3, not 2, is the appropriate place for dropping it (if it is indeed to be dropped)... and that it is not appropriate to do so in 2.x on merely a handwave by persons not involved in the previous discussions. (And of course Guido can Pronounce otherwise; I'm just saying I don't think anything less than such a pronouncement is required to remove it from 2.x. And again, I'm fine with making a clean break of it in 3.x, since setuptools is already broken there anyway.) [View Less]

4 3

Re: subprocess and EINTR errnos
by Peter Astrand July 6, 2009

July 6, 2009

On Wed, 10 Nov 2004, John P Speno wrote: Hi, sorry for the delayed response. > While using subprocess (aka popen5), I came across one potential gotcha. I've had > exceptions ending like this: > > File "test.py", line 5, in test > cmd = popen5.Popen(args, stdout=PIPE) > File "popen5.py", line 577, in __init__ > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > (on Solaris 9) > > … [View More]Would it make sense for subprocess to use a more robust read() function > which can handle these cases, i.e. when the parent's read on the pipe > to the child's stderr is interrupted by a system call, and returns EINTR? > I imagine it could catch EINTR and EAGAIN and retry the failed read(). I assume you are using signals in your application? The os.read above is not the only system call that can fail with EINTR. subprocess.py is full of other system calls that can fail, and I suspect that many other Python modules are as well. I've made a patch (attached) to subprocess.py (and test_subprocess.py) that should guard against EINTR, but I haven't committed it yet. It's quite large. Are Python modules supposed to handle EINTR? Why not let the C code handle this? Or, perhaps the signal module should provide a sigaction function, so that users can use SA_RESTART. Index: subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v retrieving revision 1.8 diff -u -r1.8 subprocess.py --- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8 +++ subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -888,6 +888,50 @@ pass + def _read_no_intr(self, fd, buffersize): + """Like os.read, but retries on EINTR""" + while True: + try: + return os.read(fd, buffersize) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + + def _read_all(self, fd, buffersize): + """Like os.read, but retries on EINTR, and reads until EOF""" + all = "" + while True: + data = self._read_no_intr(fd, buffersize) + all += data + if data == "": + return all + + + def _write_no_intr(self, fd, s): + """Like os.write, but retries on EINTR""" + while True: + try: + return os.write(fd, s) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + def _waitpid_no_intr(self, pid, options): + """Like os.waitpid, but retries on EINTR""" + while True: + try: + return os.waitpid(pid, options) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, @@ -963,7 +1007,7 @@ exc_value, tb) exc_value.child_traceback = ''.join(exc_lines) - os.write(errpipe_write, pickle.dumps(exc_value)) + self._write_no_intr(errpipe_write, pickle.dumps(exc_value)) # This exitcode won't be reported to applications, so it # really doesn't matter what we return. @@ -979,7 +1023,7 @@ os.close(errwrite) # Wait for exec to fail or succeed; possibly raising exception - data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB + data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": child_exception = pickle.loads(data) @@ -1003,7 +1047,7 @@ attribute.""" if self.returncode == None: try: - pid, sts = os.waitpid(self.pid, os.WNOHANG) + pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG) if pid == self.pid: self._handle_exitstatus(sts) except os.error: @@ -1015,7 +1059,7 @@ """Wait for child process to terminate. Returns returncode attribute.""" if self.returncode == None: - pid, sts = os.waitpid(self.pid, 0) + pid, sts = self._waitpid_no_intr(self.pid, 0) self._handle_exitstatus(sts) return self.returncode @@ -1049,27 +1093,33 @@ stderr = [] while read_set or write_set: - rlist, wlist, xlist = select.select(read_set, write_set, []) + try: + rlist, wlist, xlist = select.select(read_set, write_set, []) + except select.error, e: + if e[0] == errno.EINTR: + continue + else: + raise if self.stdin in wlist: # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), input[:512]) + bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512]) input = input[bytes_written:] if not input: self.stdin.close() write_set.remove(self.stdin) if self.stdout in rlist: - data = os.read(self.stdout.fileno(), 1024) + data = self._read_no_intr(self.stdout.fileno(), 1024) if data == "": self.stdout.close() read_set.remove(self.stdout) stdout.append(data) if self.stderr in rlist: - data = os.read(self.stderr.fileno(), 1024) + data = self._read_no_intr(self.stderr.fileno(), 1024) if data == "": self.stderr.close() read_set.remove(self.stderr) Index: test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.14 diff -u -r1.14 test_subprocess.py --- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14 +++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -7,6 +7,7 @@ import tempfile import time import re +import errno mswindows = (sys.platform == "win32") @@ -35,6 +36,16 @@ fname = tempfile.mktemp() return os.open(fname, os.O_RDWR|os.O_CREAT), fname + def read_no_intr(self, obj): + while True: + try: + return obj.read() + except IOError, e: + if e.errno == errno.EINTR: + continue + else: + raise + # # Generic tests # @@ -123,7 +134,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stdout.write("orange")'], stdout=subprocess.PIPE) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_stdout_filedes(self): # stdout is set to open file descriptor @@ -151,7 +162,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stderr.write("strawberry")'], stderr=subprocess.PIPE) - self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()), + self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)), "strawberry") def test_stderr_filedes(self): @@ -186,7 +197,7 @@ 'sys.stderr.write("orange")'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - output = p.stdout.read() + output = self.read_no_intr(p.stdout) stripped = remove_stderr_debug_decorations(output) self.assertEqual(stripped, "appleorange") @@ -220,7 +231,7 @@ stdout=subprocess.PIPE, cwd=tmpdir) normcase = os.path.normcase - self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) + self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() @@ -230,7 +241,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_communicate(self): p = subprocess.Popen([sys.executable, "-c", @@ -305,7 +316,8 @@ 'sys.stdout.write("\\nline6");'], stdout=subprocess.PIPE, universal_newlines=1) - stdout = p.stdout.read() + + stdout = self.read_no_intr(p.stdout) if hasattr(open, 'newlines'): # Interpreter with universal newline support self.assertEqual(stdout, @@ -343,7 +355,7 @@ def test_no_leaking(self): # Make sure we leak no resources - max_handles = 1026 # too much for most UNIX systems + max_handles = 10 # too much for most UNIX systems if mswindows: max_handles = 65 # a full test is too slow on Windows for i in range(max_handles): @@ -424,7 +436,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, preexec_fn=lambda: os.putenv("FRUIT", "apple")) - self.assertEqual(p.stdout.read(), "apple") + self.assertEqual(self.read_no_intr(p.stdout), "apple") def test_args_string(self): # args is a string @@ -457,7 +469,7 @@ p = subprocess.Popen(["echo $FRUIT"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_shell_string(self): # Run command through the shell (string) @@ -466,7 +478,7 @@ p = subprocess.Popen("echo $FRUIT", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_call_string(self): # call() function with string argument on UNIX @@ -525,7 +537,7 @@ p = subprocess.Popen(["set"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_shell_string(self): # Run command through the shell (string) @@ -534,7 +546,7 @@ p = subprocess.Popen("set", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_call_string(self): # call() function with string argument on Windows /Peter Åstrand <astrand(a)lysator.liu.se> [View Less]

5 5

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 6, 2009

July 6, 2009

At 07:18 PM 7/6/2009 +0200, Tarek Ziadé wrote: > >> So is PEP 376 going to be able to cope with the stuff I have installed > >> at the moment? If not, what's the point??? > > > > If I understand Tarek's proposal correctly, then no, it will not cope. > >Why that ? Can you detail ? On a system that uses only plain distutils >distributions, it'll work. Paul's question, as I understood it, was whether pkgutil would handle the existing .egg-info's. Your … [View More]

2 1

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 6, 2009

July 6, 2009

At 04:14 PM 7/6/2009 +0100, Paul Moore wrote: >2009/7/6 Nick Coghlan <ncoghlan(a)gmail.com>: > > P.J. Eby wrote: > >> At 08:43 PM 7/5/2009 +0200, Tarek Ziadé wrote: > >>> But if it's based on PEP 302 protocols and if the pkgutil code works > >>> with the sys.meta_path hook, > >>> setuptools could then provide its loader, based on its EggFormats and > >>> act as a provider without being broken. > >> > >> … [View More]You misunderstand me. The whole point of putting .egg-info in distutils > >> in the first place was to enable setuptools to detect the presence of > >> disutils-installed packages. That's what's broken by changing the name. > >This is getting confusing. Is Phillip saying that setuptools will cope >with the file changing to a directory without modification, but it >can't handle a change in the name? The existing versions of setuptools will read a file or a directory with no problem; it's the name change that will require a code change, and it's a rather more complex issue than just one name change, because it'll need to support both names. What's more, on the build/install side, it'll have to figure out whether to use the new name or the old name when creating a project's metadata for installation in single-version mode. In other words, this will likely affect pip as well, or at least the parts of setuptools that pip uses. >My site-packages has a confusing mix of egginfo directories and files. >Note that I NEVER use setuptools other than where an existing >package's setup.py requires it. In that case, I still only do python >setup.py bdist_wininst and install the generated installer. > >So is PEP 376 going to be able to cope with the stuff I have installed >at the moment? If not, what's the point??? If I understand Tarek's proposal correctly, then no, it will not cope. >If setuptools is not going to change to conform to PEP 376, then any >tools built using PEP 376 will fail to recognise my coverage install. I'm all in favor of adding RECORD support to setuptools; it was in fact my idea to have the file there in the first place. Adding a RECORD file doesn't introduce any new and weird name migration requirements, which is why I'd rather not change the extension if we can avoid it. Reading both names is painful, writing both is more so, and I'm not sure how many tools/users *besides* setuptools will be affected by a name change. > > How much information does setuptools actually need in order to tell that > > a distribution is already present? Presumably the existing .egginfo > > files generated by distutils are sufficient for that task? > >It appears so, but setuptools doesn't use (or create!!!) those files >in its own installer formats. Setuptools treats an .egg-info file as if it were a PKG-INFO file contained in an .egg-info directory. This allows it to treat distutils-supplied .egg-info files as if they were setuptools-supplied .egg-info directories containing exactly one file. [View Less]

3 2

semi-regular check that all committers are subscribed to python-committers
by Brett Cannon July 6, 2009

July 6, 2009

If you have commit privileges on Python but are not subscribed to python-committers, please let me know and I will subscribe you. Membership on python-committers is mandatory as that's where we announce branch freezes, etc. -Brett

2 4

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by P.J. Eby July 6, 2009

July 6, 2009

At 10:32 AM 7/6/2009 +0100, Paul Moore wrote: >I'm +0 on changing the name, as long as it's the *only* "do it this >way because setuptools isn't going to change" issue. Please note that I never said that. I was the one who suggested ".pydist", remember? I just don't want to have to complicate the pkg_resources directory scan to support a stupid name like "metadata", no offense to Tarek. ;-) I'd sooner not make the change at all, just because I doubt I'm the only person who has … [View More]

1 0

Mercurial migration: help needed
by "Martin v. Löwis" July 6, 2009

July 6, 2009

In this thread, I'd like to collect things that ought to be done but where Dirkjan has indicated that he would prefer if somebody else did it. So far, I have only one item: build identification. If you want to work on this, please either provide a patch (for trunk and/or py3k), or (if you are a committer) create a subversion branch. It seems that Barry and I agree that for the maintenance branches, sys.subversion should be frozen, so we need actually two sets of patches: one that removes sys.… [View More]

2 2

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
by average July 5, 2009

July 5, 2009

>> Is it really that confusing? I have never heard of anyone asking "what >> is py3k?" > > Do you read python-list? It has been asked. Also, some people seem to > think that py3k is different from python 3. Personally, I vote for keeping the "3k" for 3000 (or is it 3072?). I believe that py3k represents a ideal that hasn't been reached, despite being hoped for in python3. By keeping it, it confers the idea continual evolution *within* the language until that hypothetical … [View More]

2 1

PEP 376 - get_egginfo_files
by Paul Moore July 5, 2009

July 5, 2009

The PEP says: """ get_egginfo_files(local=False) -> iterator of paths Iterates over the RECORD entries and return paths for each line if the path is pointing a file located in the .egg-info directory or one of its subdirectory. """ Should this method really only return filenames noted in the RECORD file? Would it not be better for it to iterate over *all* files in the .egg-info directory? I understand that there shouldn't, in practice, be any files in that directory *not* mentioned in the … [View More]

2 2

PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
by Paul Moore July 5, 2009

July 5, 2009

This post basically follows on from the previous thread about PEP 376 (Changing the .egg-info structure) where it was pointed out that the PEP doesn't cover PEP 302 import hooks (except in the explicit special case of zip files). This is likely to be a fairly long posting, but I'd like to try and cover the subtleties a little, so that people can comment without having to refer back to too many documents. Comments are most definitely welcome! While I know PEP 302 reasonably well, and the zip … [View More]importer implementation, I'm not an expert on the egg-info structure, or the practical drivers behind it, so if I miss key issues because of a too-theoretical approach, I'd be grateful for corrections. The basic structure of PEP 302 imports is as follows: - scan sys.meta_path checking each finder in turn - if nothing found, scan sys.path and for each entry pass it to each element of sys.path_hooks, to get a finder to check - do the default filesystem processing on path items for which no path hook applies The first finder (if any) to recognise the module name wins (and returns a loader responsible for creating the actual module object). So it's the finders that are responsible for scanning the "filesystem" (more accurately, the namespace that finder manages). PEP 302 is entirely couched in terms of modules, packages and module names. There is no concept of a "distribution" in PEP 376 terms. This is entirely understandable, as imports don't know about distributions (see the docutils example in PEP 376 - the docutils *distribution* contains the *module* roman, but there's nothing about importing roman that requires you to know that it came from the docutils distribution). So we need to introduce a new concept, that of a distribution, into PEP 302 loaders. And the place it should be located is in the finder (which handles the "filesystem" aspects of the protocol. So we have the methods: Finder.list_distributions() Returns a list of all distribution names in the "filesystem" managed by the finder (usually one zip file, path directory, salite database, or whatever) Finder.get_metadata(distribution_name) Returns a metadata object for the given distribution (this is the PEP 376 Distribution object). One note here - get_egginfo_file should be specified as returning a *file-like object* rather than a file instance. Another note - get_egginfo_file and get_egginfo_files could just as easily be named get_metadata_file and get_metadata_files - just as meaningful, and less tied to the egg format (as well as avoiding the "egg" name, which I personally dislike :-)) Finder.uninstall(distribution_name, filter=callable, installer=name) Uninstall the given distribution. It's likely that many finders will be read-only. In that case, this function should return None. Otherwise, return a list of the "files" removed. (This may need some clarification, as many finders won't have a concept of a "file name"). I don't think anything else is needed to support PEP 376. The prototype implementation of PEP 376 could be extended to work with finders (doing the relevant sys.meta_path and sys.path_hooks searches). For the final implementation, the special casing of zip files would be replaced by an implementation of the extended finder protocol in the zipimporter object. There's almost certainly aspects missing from the above proposal. But it does have some definite advantages, above and beyond simply allowing PEP 302 importers to participate in the PEP 376 protocol. Setuptools-style eggs could be handled simply by creating a specialised finder (IIUC, they currently just use the standard zipimporter - the specialised version could subclass this) to override the metadata methods so as to cater for their specialised egg-info format. Other formats could be handled similarly. Does this sound sensible? Tarek, would you be OK with me attempting to modify your prototype to support this protocol? Are there any tests for PEP 376, so that I can confirm I haven't completely broken something? If I can, I'll knock up some simple prototype importers for non-standard examples, and see how they work with all this. Paul. [View Less]

4 16