This is a proto-proposal for including some functionality from virtualenv in
Python itself. I'm not entirely confident about what I'm proposing, so it's
not really PEP-ready, but I wanted to get feedback...
First, a bit about how virtualenv works (this will use Linux conventions;
Windows and some Mac installations are slightly different):
* Let's say you are creating an environment in ~/env/
* /usr/bin/python is *copied* to ~/env/bin/python
* This alone sets sys.prefix to ~/env/ (via existing code in Python)
* At this point things are broken because the standard library is not
available
* virtualenv creates ~/env/lib/pythonX.Y/site.py, which adds the system
standard library location (/usr/lib/pythonX.Y) to sys.path
* site.py itself requires several modules to work, and each of these modules
(from a pre-determined list of modules) is symlinked over from the standard
library into ~/env/lib/pythonX.Y/
* site.py may or may not add /usr/lib/pythonX.Y/site-packages to sys.path
* *Any* time you use ~/env/bin/python you'll get sys.prefix of ~/env/, and
the appropriate path. No environmental variable is required.
* No compiler is used; this is a fairly light tool
There are some tweaks to this that could be made, but I believe virtualenv
basically does things The Right Way. By setting sys.prefix All Tools Work
(there are some virtualenv alternatives that do isolation without setting
sys.prefix, but they typically break more often than virtualenv, or only
support a limited number of workflows). Also by using a distinct
interpreter (~/env/bin/python) it works fairly consistently and reliably
compared to techniques like an environmental variable. The one serious
alternative is what buildout (and virtualenv --relocatable) does, which is
to use the system Python and change the path at the beginning of all scripts
(it requires its own installer to accomplish this consistently).
But virtualenv is kind of a hack, and I believe with a little support from
Python this could be avoided. virtualenv can continue to exist to support
the equivalent workflows on earlier versions of Python, but it would not
exist (or would become much much simpler) on further Python versions.
The specific parts of virtualenv that are a hack that I would like to
replace with built-in functionality:
* I'd rather ~/env/bin/python be a symlink instead of copying it.
* I'd rather not copy (or symlink) *any* of the standard library.
* I'd rather site.py support this functionality natively (and in turn that
OS packagers support this when they make other modifications)
* Compiling extensions can be tricky because code may not find headers
(because they are installed in /usr, not ~/env/). I think this can be
handled better if virtualenv is slightly less intrusive, or distutils is
patched, or generally tools are more aware of this layout.
* This gets more complicated with a Mac framework build of Python, and
hopefully those hacks could go away too.
I am not sure what the best way to do this is, but I will offer at least one
suggestion (other suggestions welcome):
In my (proto-)proposal, a new binary pythonv is created. This is slightly
like pythonw.exe, which provides a Python interpreter on Windows which
doesn't open a new window. This binary is primarily for creating new
environments. It doesn't even need to be on $PATH, so it would be largely
invisible to people unless they use it.
If you symlink pythonv to a new location, it will effect sys.prefix
(currently sys.prefix is calculated after dereferencing the symlink).
Additionally, the binary will look for a configuration file. I'm not sure
where this file should go; perhaps directly alongside the binary, or in some
location based on sys.prefix.
The configuration file would be a simple set of assignments; some I might
imagine:
* Maybe override sys.prefix
* Control if the global site-packages is placed on sys.path
* On some operating systems there are other locations for packages installed
with the system packager; probably these should be possible to enable or
disable
* Maybe control installations or point to a file like distutils.cfg
I got some feedback from the Debian/Ubuntu maintainer that he would like
functionality that might be like this; for instance, if you have
/usr/bin/python2.6 and /usr/bin/python2.6-dbg, he'd like them to work
slightly different (e.g., /usr/bin/python2.6-dbg would look in a different
place for libraries). So the configuration file location should be based on
sys.prefix *and* the name of the binary itself (e.g.,
/usr/lib/python2.6/python-config-dbg.conf). I have no strong opinion on the
location of the file itself, only that it can be specific to the directory
and name of the interpreter.
In addition to all this, I think sys would grow another prefixy value, e.g.,
sys.build_prefix, that points to the place where Python was actually built
(virtualenv calls this sys.real_prefix, but that's not a very good name).
Some code, especially in distutils, might need to be aware of this to
compile extensions properly (we can be somewhat aware of these cases by
looking at places where virtualenv already has problems compiling
extensions).
Some people have argued for something like sys.prefixes, a list of locations
you might look at, which would allow a kind of nesting of these environments
(where sys.prefixes[-1] == sys.prefix; or maybe reversed). Personally this
seems like it would be hard to keep mental track of this, but I can
understand the purpose -- you could for instance create a kind of template
prefix that has *most* of what you want installed in it, then create
sub-environments that contain for instance an actual application, or a
checkout (to test just one new piece of code).
I'm not sure how this should best work on Windows (without symlinks, and
where things generally work differently), but I would hope if this idea is
more visible that someone more opinionated than I would propose the
appropriate analog on Windows.
--
Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking
Hey folks,
I'm going to write a PEP for inclusion of lib3to2 in the standard library
for 3.2 and above. Before do, though, do people have any quick thoughts
about it?
My inclination is to get it stabilized beforehand (perhaps during another
GSoC) by fleshing out the fixer that warns about backwards-incompatible
features in Python 3 and by finishing up the fix_imports2 fixer, probably
involving a rewrite.
http://bitbucket.org/amentajo/lib3to2 is where the source is hosted (there's
a separate branch for 3.1)
http://www.startcodon.com/wordpress/?cat=8 is where I blog about it.
http://pypi.python.org/pypi/3to2 is the PyPI page that has both of those
links.
--Joe Amenta
I have used pdb for several years and have always wanted a gdb-like
Ctrl-C handling: in gdb pressing Ctrl-C interrupts the program but the
execution can be resumed later by the user (while pdb will terminate
the program and throw you into postmortem debugging with no ability to
resume the execution). So I implemented this functionality as
http://bugs.python.org/issue7245.
The patch is very simple: install a SIGINT handler and when SIGINT
arrives, set the tracing. The signal handler is only activated when
pdb is run as a script. I cann't think of any disadvantages.
If this functionality is indeed useful and I am not missing some
serious side effects, would it be possible to review the patch?
Thanks,
Ilya Sandler
At 02:41 PM 2/20/2010 -0500, Ian Bicking wrote:
>Virtualenv uses copies when it can't use symlinks. Â A copy (or hard
>link) seems appropriate on systems that do not have symlinks. Â It
>would seem reasonable that on Windows it might look in the registry
>to find the actual location where Python was installed. Â Or...
>whatever technique Windows people think is best; it's simply
>necessary that the interpreter know its location (the isolated
>environment) and also know where Python is installed. Â All this
>needs to be calculated in C, as the standard library needs to be on
>the path very early (so os.symlink wouldn't help, but any C-level
>function to determine this would be helpful).
The ways pretty much boil down to:
1. Explicit per-instance configuration (either appended to the .exe
or in a file adjacent to it),
2. An implicit global search/lookup (PATH, registry, etc.)
3. A combination of the two, checking explicit configuration before implicit.
Since the virtualenv itself may need some kind of nearby
configuration anyway, putting it in that file seems to me like the
One Obvious Way To Do It.
Windows does have C-level APIs for reading and writing .ini files,
from the good old days before the registry existed. And the C-level
code might only need to read one entry prior to booting Python anyway
- a single call to the GetPrivateProfileString function, once you've
determined the name of the file to be read from.
Hello,
As I continue experimenting with advanced streams, I'm currently
beginning an important modification of io's Buffered and Text streams
(removal of locks, adding of methods...), to fit the optimization
process of the whole library.
However, I'm now wondering what the idea is behind the 3 main buffer
classes : Bufferedwriter, Bufferedreader and Bufferedrandom.
The i/o PEP claimed that the two first ones were for sequential streams
only, and the latter for all kinds of seekable streams; but as it is
implemented, actually the 3 classes can be returned by open() for
seekable files.
Am I missing some use case in which this distinction would be useful
(for optimizations ?) ? Else, I guess I should just create a
RSBufferedStream class which handles all kinds of situations, raising
InsupportedOperation exceptions whenever needed.... after all, text
streams act that way (there is no TextWriter or TextReader stream), and
they seem fine.
Also, io.open() might return a raw file stream when we set buffering=0.
The problem is that raw file streams are NOT like buffered streams with
a buffer limit of zero : raw streams might fail writing/reading all the
data asked, without raising errors. I agree this case should be rare,
but it might be a gotcha for people wanting direct control of the stream
(eg. for locking purpose), but no silently incomplete read/write operation.
Shouldn't we rather return a "write through" buffered stream in this
case "buffering=0", to cleanly handle partial read/write ops ?
regards,
Pascal
PS : if you have 3 minutes, I'd be very interested by your opinion on
the "advanced modes" draft below.
Does it seem intuitive to you ? In particular, shouldn't the "+" and "-"
flags have the opposite meaning ?
http://bytebucket.org/pchambon/python-rock-solid-tools/wiki/rsopen.html
I have noticed that deprecated stuff is still being used in the standard
Python library. When using modules that contain deprecated stuff you
get a warning, and as a mere user there isn't much you can do about that.
As a general rule, the Python standard library should not use deprecated
constructs in non-deprecated (or otherwise deprecated) modules.
The case I am running into is that mhlib uses multifile (in 2.6).
--
Sjoerd Mullender
Snippet from:
http://codereview.appspot.com/186247/diff2/5014:8003/7002
*PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is
slower than Unladen Swallow on non-numerical workloads. PyPy only
supports 32-bit x86 code generation. It has poor support for CPython
extension modules, making migration for large applications
prohibitively expensive.
That part at the very least has some sort of personal opinion
"prohibitively", while the other part is not completely true "slower
than US on non-numerical workloads". Fancy providing a proof for that?
I'm well aware that there are benchmarks on which PyPy is slower than
CPython or US, however, I would like a bit more weighted opinion in
the PEP.
Cheers,
fijal
Vlastimil Brom wrote:
> Vlastimil Brom <vlastimil.brom(a)gmail.com> added the comment:
>
> I just tested the fix for unicode tracebacks and found some possibly weird results (not sure how/whether it should be fixed, as these inputs are indeed rather artificial...).
> (win XPp SP3 Czech, Python 2.6.4)
>
> Using the cmd console, the output is fine (for the characters it can accept and display)
>
>>>> regex.findall(ur"\p{InBasicLatinĚ}", u"aé")
> Traceback (most recent call last):
> ...
> File "C:\Python26\lib\regex.py", line 1244, in _parse_property
> raise error("undefined property name '%s'" % name)
> regex.error: undefined property name 'InBasicLatinĚ'
>
> (same result for other distorted "proprety names" containing e.g. ěščřžýáíéúůßäëiöüîô ...
>
> However, in Idle the output differs depending on the characters present
>
>>>> regex.findall(ur"\p{InBasicLatinÉ}", u"ab c")
> yields the expected
> ...
> File "C:\Python26\lib\regex.py", line 1244, in _parse_property
> raise error("undefined property name '%s'" % name)
> error: undefined property name 'InBasicLatinÉ'
>
> but
>
>>>> regex.findall(ur"\p{InBasicLatinĚ}", u"ab c")
>
> Traceback (most recent call last):
> ...
> File "C:\Python26\lib\regex.py", line 1244, in _parse_property
> raise error("undefined property name '%s'" % name)
> File "C:\Python26\lib\regex.py", line 167, in __init__
> message = message.encode(sys.stdout.encoding)
> File "C:\Python26\lib\encodings\cp1250.py", line 12, in encode
> return codecs.charmap_encode(input,errors,encoding_table)
> UnicodeEncodeError: 'charmap' codec can't encode character u'\xcc' in position 37: character maps to <undefined>
>
> which might be surprising, as cp1250 should be able to encode "Ě", maybe there is some intermediate ascii step?
>
> using the wxpython pyShell I get its specific encoding error:
>
> regex.findall(ur"\p{InBasicLatinÉ}", u"ab c")
> Traceback (most recent call last):
> ...
> File "C:\Python26\lib\regex.py", line 1102, in _parse_escape
> return _parse_property(source, info, in_set, ch)
> File "C:\Python26\lib\regex.py", line 1244, in _parse_property
> raise error("undefined property name '%s'" % name)
> File "C:\Python26\lib\regex.py", line 167, in __init__
> message = message.encode(sys.stdout.encoding)
> AttributeError: PseudoFileOut instance has no attribute 'encoding'
>
> (the same for \p{InBasicLatinĚ} etc.)
>
Maybe it shouldn't show the property name at all. That would avoid the
problem.
>
> In python 3.1 in Idle, all of these exceptions are displayed correctly, also in other scripts or with special characters.
>
> Maybe in python 2.x e.g. repr(...) of the unicode error messages could be used in order to avoid these problems, but I don't know, what the conventions are in these cases.
>
>
> Another issue I found here (unrelated to tracebacks) are backslashes or punctuation (except the handled -_) in the property names, which just lead to failed mathces and no exceptions about unknown property names
>
> regex.findall(u"\p{InBasic.Latin}", u"ab c")
> []
>
In the re module a malformed pattern is sometimes treated as a literal:
>>> re.match(r"a{1,2", r"a{1,2").group()
'a{1,2'
which is what I'm trying to replicate, as far as possible.
Which characters should it accept when parsing the property name, even
if it subsequently rejects the name? I don't want it to accept every
character until it sees the closing '}'. I currently include
alphanumeric, whitespace, '&', '_' and '-'. '.' might be a reasonable
addition.
>
> I was also surprised by the added pos/endpos parameters, as I used flags as a non-keyword third parameter for the re functions in my code (probably my fault ...)
>
> re.findall(pattern, string, flags=0)
>
> regex.findall(pattern, string, pos=None, endpos=None, flags=0, overlapped=False)
>
> (is there a specific reason for this order, or could it be changed to maintain compatibility with the current re module?)
>
Oops! I'll fix that.
> I hope, at least some of these remarks make some sense;
> thanks for the continued work on this module!
>
All constructive remarks are welcome! :-)
Defaulting to UTC is not a good idea, which is why relevant methods take an
argument to specify whether to be UTC (exact details are in the patch; don't
remember exact details).
On Feb 16, 2010 4:07 PM, "Greg Ewing" <greg.ewing(a)canterbury.ac.nz> wrote:
Brett Cannon wrote:
> Issue 5094 already has a patch that is nearly complete to provide a
> default...
Are you sure it's really a good idea to default to UTC?
I thought it was considered a feature that datetime
objects are naive unless you explicitly specify a
timezone.
--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev(a)python.org
http:...
Hello,
THE PROBLEM:
I am having a problem that I have seen asked quite a bit on the web, with
little to no follow up.
The problem is essentially this. When embedding (LoadLibraryA()) the python
interpreter dll
in a non-windows application the developer must first create a console for
python to do output/input with.
I properly initialize the CRT and AllocConsole() to do this. I then
GetSTDHandle() for stdin and stdout accordingly
and open those handles with the requisite flags "read" for STDIN and "write"
for stdout. This all works great
and is then verified and tested to work by printf() and fgets(). This issue
however happens when attempting
to PyRun_InteractiveLoop() and PyRun_SimpleString(). A
PyRun_SimpleString("print 'test'") displays nothing in my
freshly allocated console window. Similarly a PyRun_InteractiveLoop(stdin,
NULL); yields nothing either even though
the line printf("testing"); directly ahead of it works just fine. Does
anyone have insight on how I can make this work
with the freshly allocated console's stdin/stdout/stderr?
SPECULATION:
That is the question, so now on to the speculation. I suspect that something
in the python runtime doesn't "get handles"
correctly for STDIN and STDOUT upon initialization. I have perused the
source code to find out exactly how this is done
and I suspect that it starts in PyInitializeEx with calls to
PySys_GetObject("stdin") and "stdout" accordingly. However I
don't actually see where this translates into the Python runtime checking
with the C-runtime for the "real" handles to STDIN and STDOUT. I dont ever
see the Python runtime "ask the system" where his handles to STDIN and
STDOUT are.
SUBSEQUENT QUESTION:
Is there anything I can do to initialize the Python interpreter (running as
a dll) pointing him at his appropriate STDIN and STDOUT
handles?
