[pypy-dev] Scrapy fails in PyPy

[Joe Hillenbrand]

Now that lxml works in PyPy, I've been excited to try Scrapy in PyPy 2.0

I've run into this issue. I'm not sure what could be happening here, but I
suspect it could be a twisted+pypy issue. I'm hoping it might look familiar
to someone.

ERROR: Error caught on signal handler: <bound method LogStats.spider_opened
of <scrapy.contrib.logstats.LogStats object at 0x0000000006f3a8e0>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 225,
in open_spider
        yield self.signals.send_catch_log_deferred(signals.spider_opened,
spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 53,
in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
134, in maybeDeferred
        result = f(*args, **kw)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: spider_opened() got 2 unexpected keyword arguments

ERROR: Error caught on signal handler: <bound method
LogStats.response_received of <scrapy.contrib.logstats.LogStats object at
0x0000000006f3a8e0>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
464, in _startRunCallbacks
        self._runCallbacks()
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 200,
in _on_success
        response=response, request=request, spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
19, in send_catch_log
        return signal.send_catch_log(*a, **kw)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 22,
in send_catch_log
        *arguments, **named)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: response_received() got 4 unexpected keyword
arguments

ERROR: Error caught on signal handler: <bound method
CoreStats.response_received of <scrapy.contrib.corestats.CoreStats object
at 0x00000000061b8d08>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
464, in _startRunCallbacks
        self._runCallbacks()
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 200,
in _on_success
        response=response, request=request, spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
19, in send_catch_log
        return signal.send_catch_log(*a, **kw)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 22,
in send_catch_log
        *arguments, **named)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: response_received() got 4 unexpected keyword
arguments

Here are the definitions for CoreStats and LogStats:

https://github.com/scrapy/scrapy/blob/0.16/scrapy/contrib/logstats.py
https://github.com/scrapy/scrapy/blob/0.16/scrapy/contrib/corestats.py

Let me know if this is a PyPy bug and I will turn it into a bug report.

Thanks
-Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121202/8656497c/attachment.html>