Mailman 3 November 2004 - Python-Dev

os.access versus os.exist
by Robert Brewer Nov. 15, 2004

Nov. 15, 2004

It would be awfully nice (on posix platforms, for my use-case) to find out whether a file is inaccessible due to permission restrictions, or due to non-existence. Poking around in posixmodule.c, it seems os.access() uses posix_access(PyObject *self, PyObject *args), which calls _access/_waccess as described here: http://msdn.microsoft.com/library/en-us/vclib/html/_crt__access.2c_._wac cess.asp Run-Time Library Reference _access, _waccess Determine file-access permission. int _access( … [View More] const char *path, int mode ); int _waccess( const wchar_t *path, int mode ); Parameters path File or directory path. mode Permission setting. Return Value Each function returns 0 if the file has the given mode. The function returns -1 if the named file does not exist or is not accessible in the given mode; in this case, errno is set as follows: EACCES Access denied: file's permission setting does not allow specified access. ENOENT Filename or path not found. ----------------- Here's the current posixmodule.posix_access (rev 2.329): static PyObject * posix_access(PyObject *self, PyObject *args) { char *path; int mode; int res; #ifdef Py_WIN_WIDE_FILENAMES if (unicode_file_names()) { PyUnicodeObject *po; if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) { Py_BEGIN_ALLOW_THREADS /* PyUnicode_AS_UNICODE OK without thread lock as it is a simple dereference. */ res = _waccess(PyUnicode_AS_UNICODE(po), mode); Py_END_ALLOW_THREADS return(PyBool_FromLong(res == 0)); } /* Drop the argument parsing error as narrow strings are also valid. */ PyErr_Clear(); } #endif if (!PyArg_ParseTuple(args, "si:access", &path, &mode)) return NULL; Py_BEGIN_ALLOW_THREADS res = access(path, mode); Py_END_ALLOW_THREADS return(PyBool_FromLong(res == 0)); } ----------------- It seems to me that, if EACCES and ENOENT were made available (in a future version of Python), I could solve my problem easily. One way would be to raise OSError within os.access if the result of _waccess == ENOENT, but this would cause problems with old code which depends upon the old behavior. Can we add an optional "error_if_not_found" arg to os.access? Return the result of _waccess instead of True/False? Make a new os function with the desired behavior? The next issue, then, is other platforms: is similar functionality available in OS/2, unix, etc.? Finally, if anyone has any immediate workarounds for me ;) answer in private or on c.l.p. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org [View Less]

4 3

Int literals and method calls
by Kay Schluehr Nov. 15, 2004

Nov. 15, 2004

Hi , i wondered why string and bool literals have access methods and one can call simply >>> "1".__class__ <type 'str'> >>> False.__class__ <type 'bool'> But this won't be true for int literals and float literals except for those float literals that are terminated by a dot: >>> 1..__class__ <type 'float'> The expression below raises an error: >>> 1.__class__ Traceback ( File "<interactive input>", line 1 1.__class__ … [View More]

7 10

RE: [Python-Dev] os.access versus os.exist
by Robert Brewer Nov. 15, 2004

Nov. 15, 2004

Martin v. Löwis wrote: > Robert Brewer wrote: > > It would be awfully nice (on posix platforms, for my > > use-case) to find out whether a file is inaccessible > > due to permission restrictions, or due to non-existence. > > That's easy enough: just check the exception you get when you try > opening the file. Brilliant! I'll do that. I didn't think of open() because I don't need to open it--just see if it's there. /*bonk* self Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

RE: [Python-Dev] os.access versus os.exist
by Robert Brewer Nov. 15, 2004

Nov. 15, 2004

I wrote: > > It would be awfully nice (on posix platforms, for my > > use-case) to find out whether a file is inaccessible > > due to permission restrictions, or due to non-existence. Guido van Rossum replied: > Why can't you solve this by doing a stat() when access() > returns False? In my current case, stat() gives an error (even though the file exists): OSError: [Errno 2] No such file or directory: u'S:/Scans/Projects/2004/TJ/04TJ0267.jpg' Running the same stat()… [View More]

1 0

Re: [Python-Dev] Int literals and method calls
by "Martin v. Löwis" Nov. 15, 2004

Nov. 15, 2004

Kay Schluehr wrote: > Thank You. I agree completely with Your consistency rule, which is actually > a disambiguation. On the other hand It's not just me who was confused about > method calls on int-literals and for convenience it should be > emphasized, at least > in the language reference but perhaps from the beginning, in the tutorial. Patches to the language reference are welcome. I'm not so sure about patches to the tutorial - this is a fairly infrequent problem, so the … [View More]

1 0

parse xml
by Mike Tuller Nov. 14, 2004

Nov. 14, 2004

I am just starting with python. I have a problem where I want to take a document that is in xml and take the items and enter them into a mysql database. The database part I know, I just need some hints on where to look to be able to parse something like this. <dict> <key>_name</key> <string>VLC</string> <key>info</key> … [View More]

3 2

proposal+patch: sys.gettickeraccumulation()
by Ralf W. Grosse-Kunstleve Nov. 14, 2004

Nov. 14, 2004

Proposal: >>> sys.gettickeraccumulation.__doc__ 'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.' Target environments: Python-based systems with (many) extension modules. Motivation: A fast, easy, and non-intrusive method for estimating how much time is spent in Python code and how much time in extension modules. E.g. if numarray is used it is often difficult to estimate how much time is spent in the numarray extension and … [View More]how much in application-specific Python code. Is it worth investing time reimplementing the application-specific code in C or C++? If most of the time is already spent in the numarray extension the answer will be no. Method: Determination of "micro-seconds per tick:" time.time() / sys.gettickeraccumulation() * 1.e6 A "tick" is the processing of one "Python virtual instruction." See also "setcheckinterval" here: http://www.python.org/doc/current/lib/module-sys.html A pure Python program will spent the bulk of the time interpreting bytecode. This will lead to a small value for "micro-seconds per tick." In contrast, if almost all the time is spent in extension modules, a calculation may run a long time without increasing sys.gettickeraccumulation(). For example: time ticks time/tick in micro-seconds 0.00% Python 0.008 20 405.908 10.00% Python 0.072 444680 0.163 20.00% Python 0.138 889340 0.155 30.00% Python 0.203 1334000 0.152 ... 100.00% Python 0.659 4446620 0.148 With higher resolution: time ticks time/tick in micro-seconds 0.00% Python 6.758 20 337893.748 0.10% Python 0.728 444680 1.638 0.20% Python 0.795 889340 0.894 0.30% Python 0.837 1334000 0.627 0.40% Python 0.923 1778660 0.519 0.50% Python 0.978 2223320 0.440 0.60% Python 1.091 2667980 0.409 0.70% Python 1.111 3112640 0.357 0.80% Python 1.191 3557300 0.335 0.90% Python 1.236 4001960 0.309 1.00% Python 1.315 4446620 0.296 In real applications we observed times/tick of around 10 on the same platform, indicating that significant performance increases could only be achieved through extensive recoding in a compiled language. On the other hand if we see values smaller than 1 we know that significant performance increases are achievable with a reasonable effort. Implementation: A full patch based on Python-2.4b2 is attached. The essence is this additional line in Python/ceval.c: + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; Note that _Py_Ticker and _Py_CheckInterval exist already in the Python distribution. The impact of the additional code on the runtime performance of a pure Python program is minute. On Xeon/Linux the factor is smaller than 1.00005. The factor is even smaller if extension modules are used. The full patch, the patched files, and a complete Python distribution including the patched files can be found here: http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patch http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patched_files.tar.gz http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker.tar.gz diff -u -r Python-2.4b2/Include/ceval.h Python-2.4b2_ticker/Include/ceval.h --- Python-2.4b2/Include/ceval.h 2004-10-10 19:40:35.000000000 -0700 +++ Python-2.4b2_ticker/Include/ceval.h 2004-11-12 18:16:28.000000000 -0800 @@ -68,6 +68,11 @@ /* this used to be handled on a per-thread basis - now just two globals */ PyAPI_DATA(volatile int) _Py_Ticker; +#ifndef HAVE_LONG_LONG +PyAPI_DATA(volatile long) _Py_TickerAccumulation; +#else +PyAPI_DATA(volatile PY_LONG_LONG) _Py_TickerAccumulation; +#endif PyAPI_DATA(int) _Py_CheckInterval; /* Interface for threads. diff -u -r Python-2.4b2/Objects/longobject.c Python-2.4b2_ticker/Objects/longobject.c --- Python-2.4b2/Objects/longobject.c 2004-09-19 23:14:54.000000000 -0700 +++ Python-2.4b2_ticker/Objects/longobject.c 2004-11-12 18:20:53.000000000 -0800 @@ -38,6 +38,7 @@ #define SIGCHECK(PyTryBlock) \ if (--_Py_Ticker < 0) { \ + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; \ _Py_Ticker = _Py_CheckInterval; \ if (PyErr_CheckSignals()) { PyTryBlock; } \ } diff -u -r Python-2.4b2/PC/os2emx/python24.def Python-2.4b2_ticker/PC/os2emx/python24.def --- Python-2.4b2/PC/os2emx/python24.def 2004-10-10 19:40:50.000000000 -0700 +++ Python-2.4b2_ticker/PC/os2emx/python24.def 2004-11-12 18:16:47.000000000 -0800 @@ -743,6 +743,7 @@ "_Py_CheckRecursionLimit" "_Py_CheckInterval" "_Py_Ticker" + "_Py_TickerAccumulation" ; From python24_s.lib(compile) "PyCode_New" diff -u -r Python-2.4b2/Python/ceval.c Python-2.4b2_ticker/Python/ceval.c --- Python-2.4b2/Python/ceval.c 2004-10-10 19:40:50.000000000 -0700 +++ Python-2.4b2_ticker/Python/ceval.c 2004-11-12 18:22:55.000000000 -0800 @@ -373,6 +373,7 @@ pendinglast = j; _Py_Ticker = 0; + _Py_TickerAccumulation = 0; things_to_do = 1; /* Signal main loop */ busy = 0; /* XXX End critical section */ @@ -476,6 +477,11 @@ per thread, now just a pair o' globals */ int _Py_CheckInterval = 100; volatile int _Py_Ticker = 100; +#ifndef HAVE_LONG_LONG +volatile long _Py_TickerAccumulation = 0; +#else +volatile PY_LONG_LONG _Py_TickerAccumulation = 0; +#endif PyObject * PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals) @@ -776,6 +782,7 @@ a try: finally: block uninterruptable. */ goto fast_next_opcode; } + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; _Py_Ticker = _Py_CheckInterval; tstate->tick_counter++; #ifdef WITH_TSC diff -u -r Python-2.4b2/Python/sysmodule.c Python-2.4b2_ticker/Python/sysmodule.c --- Python-2.4b2/Python/sysmodule.c 2004-08-12 11:19:17.000000000 -0700 +++ Python-2.4b2_ticker/Python/sysmodule.c 2004-11-12 18:51:14.000000000 -0800 @@ -442,6 +442,20 @@ "getcheckinterval() -> current check interval; see setcheckinterval()." ); +static PyObject * +sys_gettickeraccumulation(PyObject *self, PyObject *args) +{ +#ifndef HAVE_LONG_LONG + return PyInt_FromLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker); +#else + return PyLong_FromLongLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker); +#endif +} + +PyDoc_STRVAR(gettickeraccumulation_doc, +"gettickeraccumulation() -> current count of bytecodes processed by the interpreter." +); + #ifdef WITH_TSC static PyObject * sys_settscdump(PyObject *self, PyObject *args) @@ -763,6 +777,8 @@ setcheckinterval_doc}, {"getcheckinterval", sys_getcheckinterval, METH_NOARGS, getcheckinterval_doc}, + {"gettickeraccumulation", sys_gettickeraccumulation, METH_NOARGS, + gettickeraccumulation_doc}, #ifdef HAVE_DLOPEN {"setdlopenflags", sys_setdlopenflags, METH_VARARGS, setdlopenflags_doc}, [View Less]

4 3

Re: [Python-Dev] syntactic shortcut - unpack to variably sized list
by Nick Coghlan Nov. 14, 2004

Nov. 14, 2004

David Wilson wrote: > I dislike the use of 'popslice' as it implies somewhere along the lines > the use of the slice type, which isn't happening. If I was writing it, using a slice object as an argument to PySequence_GetItem is *exactly* the method I would be using to implement it: indices = PySlice_New(start, end, step); x = PySequence_GetItem(L, indices); PySequence_DelItem(L, indices); return x; Add reference counting, error checking and actually hook this up to the … [View More]list type in the interpreter, and you're basically done. Heck, it could be argued that list.pop should just accept slices for the argument: a, b = L.pop(slice(2)) # Pop the first two items x, y, z = L.pop(slice(1:6:2)) # Pop items 1, 3 & 5 No new method required in that case - if you provide an index to list.pop, it will return a single item, if you provide a slice object, it will return a list. (Similar to the way list.__getitem__ currently works) > Also, what if at some > future date it is decided that all magical syntax should be exposed > using friendly method names too? Then you might have a list.slice which > work in a much different way.[1] List slicing is already exposed in Python in a few ways: L[:2] L.__getitem__(slice(2)) operator.itemgetter(slice(2))(L) Adding yet another way to spell it doesn't seem too likely. Given that lists *had* a __getslice__ method, and it has since been deprecated in favour of slice() objects, I'd be especially surprised to see list.slice (or equivalent) making a comeback. I will note that I only looked into the history of slicing this evening (when checking the C API for the sample implementation above). Having done so, the trend seems to have been away from having special methods for handling slices (__getslice__, __setslice__, __delslice__), and towards methods which can handle either a numeric index or a slice object (__getitem__, __setitem__, __delitem__). So maybe "list.pop(slice(2))" is the way to go (a non-optimised version could be written fairly easily using the existing internal list functions to retrieve and then delete the data. The main trick would be to leave the standard path through listpop alone to avoid any performance hit to the current users of list.pop() and list.pop(x)). > Second of all, you cannot have a non-default argument follow a default > argument, so your proposed syntax would not work without breaking > standard semantics and writing special code to act differently depending > on the argument list length, and also take into account if keyword args > are used, etc, etc. The signature I proposed is identical to the existing signatures for the builtins range(), xrange() and slice(). It's true that normal Python functions and methods can't have a signature like that (it's a syntax error). C extensions (including the CPython interpreter) can do whatever they want, though. In this case, I think being consistent with slice() is more important than being consistent with normal function signatures. However, the suggestion above (allowing slice objects as arguments to list.pop) takes advantage of the existing slice() builtin, so none of that non-standard argument handling would need to be duplicated. > For an extra method, +1, but with a different name (I was thinking > 'popitems', but dict.popitem returns a (k, v)). What do you think of allowing slice objects to be used with list.pop()? > Oh one last thought. If "*<identifier>" were to be added, then what of > "**<identifier>"? I can't think of any use for it, and so the whole > scheme might appear confusing to beginners who find they can use ** in > function declarations, but not in expressions. Aye, I don't think it's an accident that "a, b *c = L" has been knocked on the head 3 times now. My main gripe with it is that it requires new syntax, yet still only supports a small subset of slicing. My idea about enhancing list.pop or adding a new method to list to provide the desired functionality is aimed at avoiding having the discussion *again* in the future (since clearly this is functionality some portion of the user community would make use of). I don't actually need the functionality myself, so I doubt I'll get around to providing a sample implementation. But the suggestion is there if any of the people that *do* want the feature would like to implement it. > [1] Exposing magical syntax (eg. indexers) as real methods is the > Microsoft way for .NET (or at least, according to a C# book I read). It > proves useful in occasions where interoperability with lesser systems is > required, for example, a language that does not support custom indexers, > or for enabling clients of an object bridge [that does not support > cust..] to make full use of your classes. Python already exposes most things as methods - I found browsing through the Language Reference a few months back extremely enlightening in that regard (specifically Section 3.3 Special Method Names). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan(a)email.com | Mobile: +61 409 573 268 [View Less]

1 0

Re: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
by Ralf W. Grosse-Kunstleve Nov. 13, 2004

Nov. 13, 2004

> it should go in after 2.4 > final is out, and the release24-maint branch has been cut. OK, I'll wait until 2.4 is out and submit another patch based on the mainline CVS through the SourceForge system. > if this were to go into Python - What speaks against making the ticker accumulation available at the script level? - Is there an alternative way of achieving the goal of my proposal? It was: > A fast, easy, and non-intrusive method for estimating how much time > is … [View More]

1 0

Minor cookielib changes for 2.4b1
by John J Lee Nov. 13, 2004

Nov. 13, 2004

Would anyone care to check this cookielib.py patch in before 2.4b1? http://python.org/sf/1028908 Quote from above: The patch contains uncontroversial changes to cookielib and associated modules, documentation and tests. Would be unfortunate not to have these in 2.4.0. [...SNIP...] John

4 5