Hello! Nobody noted the message in c.l.py, let me try to ask you before I file a bug report on SF. ----- Forwarded message from Oleg Broytmann <phd@phd.pp.ru> ----- On Thu, Jun 27, 2002 at 08:45:16PM +0000, Bengt Richter wrote:
Recently Python (one program that I am dbugging) started to crash. FreeBSD kills it with "Bus error", Linux with "Segmentation fault".
I think the program crashed in the cPickle.dump(file, 1)
I replaced cPikle.dump with pikle.dump and got infinite rcursion. The traceback is below. What's that? Are there any limits that an object to be pikled must follow? Could it be a tree with loops? (I am pretty sure it could - I used the program for years, and data structures was not changed much). Could it be "new" Python class? (Recently I changed one of my classes to be derived from builtin list instead of UserList). Well (or not so well), the traceback: Traceback (most recent call last): File "/home/phd/lib/bookmarks_db/check_urls.py", line 158, in ? run() File "/home/phd/lib/bookmarks_db/check_urls.py", line 145, in run storage.store(root_folder) File "bkmk_stpickle.py", line 23, in store File "/usr/local/lib/python2.2/pickle.py", line 973, in dump Pickler(file, bin).dump(object) File "/usr/local/lib/python2.2/pickle.py", line 115, in dump self.save(object) File "/usr/local/lib/python2.2/pickle.py", line 219, in save self.save_reduce(callable, arg_tup, state) File "/usr/local/lib/python2.2/pickle.py", line 245, in save_reduce save(arg_tup) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 374, in save_tuple save(element) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) [about 1000 lines skipped - they are all the same] File "/usr/local/lib/python2.2/pickle.py", line 498, in save_inst save(stuff) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 447, in save_dict save(value) File "/usr/local/lib/python2.2/pickle.py", line 219, in save self.save_reduce(callable, arg_tup, state) File "/usr/local/lib/python2.2/pickle.py", line 245, in save_reduce save(arg_tup) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 374, in save_tuple save(element) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 414, in save_list save(element) File "/usr/local/lib/python2.2/pickle.py", line 143, in save pid = self.persistent_id(object) RuntimeError: maximum recursion depth exceeded ----- End forwarded message ----- Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
[Oleg Broytmann]
Nobody noted the message in c.l.py, let me try to ask you before I file a bug report on SF.
I saw the c.l.py msgs but found nothing to say: before you reduce this to a program someone else can run and get the same error, it's going to remain a mystery. Mysteries belong on SF, though. You may want to see whether the problem persists with a build from current CVS Python (something someone would have tried last week if they had a program they could run).
On Sun, Jun 30, 2002 at 04:06:36PM -0400, Tim Peters wrote:
I saw the c.l.py msgs but found nothing to say: before you reduce this to a program someone else can run and get the same error, it's going to remain a
I think I can reduce this, but I am afraid the data structure still will be large,
mystery. Mysteries belong on SF, though. You may want to see whether the
That what I don't want to do - file a mysterious bug report.
problem persists with a build from current CVS Python (something someone would have tried last week if they had a program they could run).
I'll try. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
[Oleg Broytmann]
I think I can reduce this, but I am afraid the data structure still will be large,
That doesn't matter. It's the amount of *code* we don't understand and have to learn that matters. If you could reduce this to a gigabyte of pickle input that we only need to feed into pickle, that would be great.
That what I don't want to do - file a mysterious bug report.
That's what bug reports are best for! Now you've got comments about your bug scattered across comp.lang.python and python-dev, and nobody will be able to find them again. Attaching new info to a shared bug report is much more effective. if-there-isn't-a-mystery-there-isn't-a-bug<wink>-ly y'rs - tim
On Sun, Jun 30, 2002 at 04:32:35PM -0400, Tim Peters wrote:
[Oleg Broytmann]
I think I can reduce this, but I am afraid the data structure still will be large,
That doesn't matter. It's the amount of *code* we don't understand and have to learn that matters. If you could reduce this to a gigabyte of pickle input that we only need to feed into pickle, that would be great.
Ok. From today I have a lot of spare time and very good almost free Internet connection, so I can investigate things. I can post the results of my investigation to the developers list or to the c.l.py, if anyone is interested.
That what I don't want to do - file a mysterious bug report.
That's what bug reports are best for!
Hmm, I thought they are not, as those mysterious bug reports take up space and time - someone have to read it, at least; but they are not help in any way.
Now you've got comments about your bug scattered across comp.lang.python and python-dev, and nobody will be able to find them again. Attaching new info to a shared bug report is much more effective.
Ah, I see now. I am strictly attached to email and email archives, and I am always hating web-based collaboration tools, but you made a good point. Still, life is too short to spend it in the SF slooow interface :( Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
[Oleg Broytmann]
On Sun, Jun 30, 2002 at 04:32:35PM -0400, Tim Peters wrote:
Attaching new info to a shared bug report is much more effective.
You know, it is only effective when it works! The SF tracker did not work for me. I filed a bug, and checked it was correctly saved (through tedious paging all over). Then, much later, I received a message from a maintainer saying that my report was empty. It surely was not after I filed it. I guess the SF tracker works only for those using it very often :-).
Ah, I see now. I am strictly attached to email and email archives, and I am always hating web-based collaboration tools, but you made a good point. Still, life is too short to spend it in the SF slooow interface :(
Slow, hardly usable, and not even dependable. Email might be less black-holish, after all. Moreover, most people (maintainers included) know how to read and file an email. I've a hard time believing people who tell me that maintainers are unable to sort emails without loosing them, or that I can really sort their own email better than they can. I usually praise maintainers as intelligent people. :-) -- François Pinard http://www.iro.umontreal.ca/~pinard
François Pinard wrote:
Slow, hardly usable, and not even dependable. Email might be less black-holish, after all. Moreover, most people (maintainers included) know how to read and file an email.
might so be, but such archives are not shared.
I usually praise maintainers as intelligent people. :-)
so why not do as we tell you, and post bug reports on SF? or better, join the roundup team, and make sure it's good enough to replace the SF tracker. (as far as I know, it already is -- but someone still needs to set it up, write a script that pulls all data out of the old system, etc). </F>
[Fredrik Lundh]
I usually praise maintainers as intelligent people. :-)
so why not do as we tell you, and post bug reports on SF?
Because this is not an efficient way to proceed, and does not always work. I do not have much spare time, and would hate seeing it spoiled, fighting with artificial problems coming from tools doomed to be replaced anyway.
or better, join the roundup team, and make sure it's good enough to replace the SF tracker.
I found the SF tracker so unattractive that I've been tempted to do that indeed. Yet, thinking more about it, it is non-sense for me to invest vast amount of energies merely to acquire the capability of submitting numerous little things (as documentation nits, for example). A long while ago, I witnessed that we had to pay real money to machine constructors, yearly, for having the right of submitting reports to them in such a way that we could later use their consequent works. The free software movement turned the values around, and refreshingly underlined that reporting a problem is a contribution from the user to the software maintainer and indirectly, to the community. For many years, we are living a progressive swing-back, in which expenditure of money has been replaced by all the stunts and sufferings induced by inadequate communication tools, like bug trackers. The price to pay is rather high. If users' contributions were really welcome, maintainers would not try to force users into this. It is much easier and comfortable for me to be a mere user, and let others pay the price. However, my principles and education strongly tell me that when something is given to me (like Python), it is only normal and natural trying to give something back. As I contributed many thousands of hours for other projects, I did my share overall, and my own principles are satisfied. Enough for me to refuse a high price ticket, in free time and irritation, before I could offer my work or dedication. Oh, I may come to like bug trackers. But surely, I find extremely distasteful being forced into them. -- François Pinard http://www.iro.umontreal.ca/~pinard
François, I swear you spend more time complaining about SF than it would take to just use it. You're not going to prevail on this, so please save everyone's time (yours and ours) by skipping the repetition.
... The free software movement turned the values around, and refreshingly underlined that reporting a problem is a contribution from the user to the software maintainer and indirectly, to the community.
Problem reports are certainly appreciated. Problem reports via one-to-one email works great for a new open source project with users numbering in the dozens, but it doesn't scale. Python has hundreds of thousands of users now, and more reports than the sum total of developers can handle. Any project of this size has to change how it works. Guido held on to his Guido's-inbox bug reporting system for a year after it totally broke down, and it took a lot of extra work to recover from the chaos it fell into. Despite its flaws, the SF-based trackers work at least a thousand times better than that did in the end. We can't go back.
[Tim Peters]
François, I swear you spend more time complaining about SF than it would take to just use it.
So far, its kif-kif. (I'm not sure it is English: I mean that I spent about the same time complaining that I spent trying to use the beast). The balance would probably break if I was trying to use SF more often! :-)
You're not going to prevail on this, so please save everyone's time (yours and ours) by skipping the repetition.
I'm not at all trying to prevail, I do not have such needs. However, it is worth underlining that there are other ways to communication, and that the Python community might disserve itself by asserting there is only one.
Despite its flaws, the SF-based trackers work at least a thousand times better than that did in the end. We can't go back.
There is only once choice left, then, and that's going forward! I intend to give `roundup' an honest try, while understanding it is still in the works. -- François Pinard http://www.iro.umontreal.ca/~pinard
On 1 Jul 2002 at 15:15, Fredrik Lundh wrote: [complaints about SF bug tracker]
or better, join the roundup team, and make sure it's good enough to replace the SF tracker. (as far as I know, it already is -- but someone still needs to set it up, write a script that pulls all data out of the old system, etc).
Someone is setting it up, and has written those scripts. A working demo (populated with PythonLabs history) should soon be available. I have to say that my "good enough" has been focussed on funtionality more than usability. I've never noticed any particular[1] difficulty in using the SF tracker to enter or post info on a bug, so François probably has a different "good enough" :-). -- Gordon http://www.mcmillan-inc.com/ [1] Writing an app as a set of cgi's is a fast way to write a mediocre GUI. Writing a *good* GUI in this environment is extremely difficult and ugly, because you end up with lots of just-barely-portable- by-any-definition-thereof javascript, and Aahz is left out in the cold. Oh well, at least no one had a requirement that it be usable throught their cell phone...
On Mon, Jul 01, 2002, Gordon McMillan wrote:
I have to say that my "good enough" has been focussed on funtionality more than usability. I've never noticed any particular[1] difficulty in using the SF tracker to enter or post info on a bug, so François probably has a different "good enough" :-).
[1] Writing an app as a set of cgi's is a fast way to write a mediocre GUI. Writing a *good* GUI in this environment is extremely difficult and ugly, because you end up with lots of just-barely-portable- by-any-definition-thereof javascript, and Aahz is left out in the cold. Oh well, at least no one had a requirement that it be usable throught their cell phone...
If it's usable in Lynx, it should be usable on a cell phone. Anyway, I find it difficult to believe that you're having a lot of trouble writing a good GUI with plain HTML, at least by the standards of people here -- a clean, accessible, and functional interface *is* a good GUI. If you've got an URL for testing, I'd be glad to give feedback (and I'll even be willing to fire up Konquerer to cross-check). I'm curious whether you think that Google Groups Advanced Search is a good GUI. What about the Google Groups thread view? Finally, it takes some effort, but it's not *that* hard to use JavaScript that degrades gracefully when JavaScript isn't available. For more info, see http://www.rahul.net/aahz/javascript.html -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/
On 01 July 2002, Gordon McMillan said:
[1] Writing an app as a set of cgi's is a fast way to write a mediocre GUI.
Rumour has it that there are several fine web application frameworks available for Python. I'm partial to Quixote [1] myself, but I've also heard good things about WebWare. Greg [1] http://www.mems-exchange.org/software/quixote/ -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ All programmers are playwrights and all computers are lousy actors.
On Tue, 2 Jul 2002, Greg Ward wrote:
On 01 July 2002, Gordon McMillan said:
[1] Writing an app as a set of cgi's is a fast way to write a mediocre GUI.
Rumour has it that there are several fine web application frameworks available for Python. I'm partial to Quixote [1] myself, but I've also heard good things about WebWare.
I think the point was that web-based GUIs tend to be rather mediocre, regardless of which toolkit is used. To some degree I have to agree -- you typically end up with a very clunky GUI with lots of high latency hits to a server for updates, or a very complex frontend implemented with a large and difficult to maintain body of Javascript. Some progress has been made to improve the situation, although the state-of-the-art is far from ideal. We can take this discussion off python-dev if anyone wants to know more about my thoughts on this matter. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
[Kevin Jacobs]
I think the point was that web-based GUIs tend to be rather mediocre, regardless of which toolkit is used. To some degree I have to agree -- you typically end up with a very clunky GUI with lots of high latency hits to a server for updates, or a very complex frontend implemented with a large and difficult to maintain body of Javascript.
Some progress has been made to improve the situation, although the state-of-the-art is far from ideal.
We can take this discussion off python-dev if anyone wants to know more about my thoughts on this matter.
I'd be interested to hear more. There's a current discussion on comp.lang.python where you may want to post your thoughts: Here are two different pointers to the beginning of today's installments: http://groups.google.com/groups?selm=3d21259f%240%2428006%24afc38c87%40news. optusnet.com.au http://mail.python.org/pipermail/python-list/2002-July/111256.html Cheers, // mark -
[Oleg Broytmann]
Ok. From today I have a lot of spare time and very good almost free Internet connection, so I can investigate things.
Great!
I can post the results of my investigation to the developers list or to the c.l.py, if anyone is interested.
It's off-topic on Python-Dev, and it will get ignored on c.l.py (you already tried that -- what changed since the last time you got ignored <wink>?). Attach info to a bug report -- that's where it belongs.
... Still, life is too short to spend it in the SF slooow interface :(
That's a feature: it encourages people to add only focused comments of real value <0.9 wink>.
Is this posted as an SF bug report yet? I have a small program which can hit the recursion limit in pickle and cause sig11 in cPickle. It's a very deep nested tuple in this case. If the first 'pickle.dump()' call is not commented out, the program dies with "RuntimeError: maximum recursion depth exceeded". If the second bit of code is executed, it dies with segmentation violation. On my system, redhat 7.2, the stack in the main thread is very large, but the stack in other threads is very small. On systems where the main stack is smaller, just running cPickle.dump(x, open("/dev/null", "w")) should show the problem, no threads needed. Probably some sort of stack check should be present in cPickle, but there's nothing much that can be done about data structures that are so deeply recursive that they fill the stack. Well, pickle could be rewritten to be iterative, but that's a tall order. (traceback from cPickle: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1026 (LWP 1198)] 0x4004912c in __new_sem_post (sem=0x8154358) at semaphore.c:137 137 semaphore.c: No such file or directory. in semaphore.c (gdb) where #0 0x4004912c in __new_sem_post (sem=0x8154358) at semaphore.c:137 #1 0x08095ec2 in PyThread_release_lock (lock=0x8154358) at Python/thread_pthread.h:412 #2 0x08079a4d in PyEval_SaveThread () at Python/ceval.c:329 #3 0x4025b09d in write_file (self=0x80ff3e8, s=0x4025db54 "(", n=1) at /home/jepler/cvs/python/dist/src/Modules/cPickle.c:414 #4 0x4025396a in save_tuple (self=0x80ff3e8, args=0x4062a66c) at /home/jepler/cvs/python/dist/src/Modules/cPickle.c:1340 #5 0x40254f18 in save (self=0x80ff3e8, args=0x4062a66c, pers_save=0) at /home/jepler/cvs/python/dist/src/Modules/cPickle.c:1944 #6 0x402539b1 in save_tuple (self=0x80ff3e8, args=0x4062a68c) at /home/jepler/cvs/python/dist/src/Modules/cPickle.c:1350 ...) import pickle, cPickle x = () for i in range(100000): x = (x,) #pickle.dump(x, open("/dev/null", "w")) import thread, time thread.start_new_thread(cPickle.dump, (x, open("/dev/null", "w"))) time.sleep(1000)
On Mon, Jul 01, 2002 at 06:46:44AM -0500, jepler@unpythonic.net wrote:
Is this posted as an SF bug report yet?
Not yet.
I have a small program which can hit the recursion limit in pickle and cause sig11 in cPickle. It's a very deep nested tuple in this case.
In my case the data strucrure is more complex, but less deep. It is a tree of objects (about 3000 objects). I tried to create lesser trees, but the bug disappeared. One interesting thing to note is that when I changed builtin list back to UserList the problem disappeared. That is, my problem is related to pickling new classes. I narrowed the code to just 180 lines. The problem manifests itself after loading initial tree, running inverse linker, and saving data back. Inverse linker runs over all objects in the tree and adds a link to its parent to every object. So I think the bug is in the pickling a data structure with loops. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
Bug reports are off-topic on Python-Dev, unless the Python developers need this list to collaborate on an implementation change. Please keep bug reports on SourceForge. Like it or not, putting information in an SF bug report is only way your bug has a chance to get addressed.
participants (10)
-
Aahz -
Fredrik Lundh -
Gordon McMillan -
Greg Ward -
jepler＠unpythonic.net -
Kevin Jacobs -
Mark McEahern -
Oleg Broytmann -
pinard＠iro.umontreal.ca -
Tim Peters