[python-nl] Error in python multiprocessing!

[Guido Wesdorp]

Hi there,

Seems like Martin is right, the error seems to be that your object isn't
serializable, and it indeed seems to be 'pickle' that's used for
serialization (though I'm not 100% sure). The _io.TextIOWrapper you're
trying to queue is probably a 'foreign' object, implemented in C or C++ or
sth, and can therefore not be serialized. A solution would be to store your
data in a simpler structure such as a dict or a simple object and add that
to the queue instead.

Hope this helps.

Cheers,

Guido

On Sat, Sep 3, 2016 at 9:02 PM, Martin Fitzpatrick <
martin.fitzpatrick at gmail.com> wrote:

> Hi Akshay
>
> It's a little difficult to understand what you're doing, but as a general
> rule I would avoid using both multiprocessing and Qt threads (if only
> because this is going to confuse things). The error you're receiving looks
> like it is coming from the serialisation (pickle?) of the data in/out of
> the task.
>
> For a problem like this I would suggest sticking with Qt threads and
> implementing something via QRunnable/QThreadpool. I have a brief (draft)
> writeup of this process on my site at: http://mfitzp.io/
> application-multithreading-with-pyqt/  ...will be finished up in a few
> weeks.
>
> The approach is to create a QRunnable class that accepts a function to run
> and passes notification/results out as signals. To start a job (e.g.
> process a file) you would pass in your function, the filename and then
> attach a listener to the 'result' signal. Once the task is run your data
> will come back out via that. You can for example attach this signal to a
> slot (python function) to update your GUI.
>
> If the plotting is causing a slow down, you might also want to try
> PyQtGraph.
>
> Hope that helps
>
> Martin
>
>
>
> On 27 August 2016 at 10:16, Akshay Verma via Python-nl <
> python-nl at python.org> wrote:
>
>> Hi all,
>>
>> After a lot of searching on the internet, I am looking for help from you
>> guys.
>>
>> I am working on a Qt based GUI which takes in a stream of file paths
>> coming from a separate process and reads those data file for
>> post-processing and plotting. Since post-processing and plotting have
>> become a bit involved, there is a request to improve the performance of the
>> GUI. Currently the GUI takes in the input message with the path of the file
>> and gives a SIGNAL to the Worker thread (Qt thread) to process the file.
>> The processing results are saved in the object for reference and display
>> within the GUI. Simple separation of the threads does not help a lot for
>> intensive processing cases. I had to add a sleep of 1 sec to quick fix the
>> GIL of Python and make the GUI responsive. To improve the solution, I
>> thought of using Multiprocessing for the post-processing task. I am trying
>> to add the task to queue and get back the result in a queue. When I
>> implement this solution, I get '*TypeError: cannot serailize
>> '_io.TextIOWrapper' object'* at the call of start() of the Process.
>>
>> What could be the cause of this error? Is there a better way to solve
>> this issue? Any suggestions are welcome.
>>
>> Regards, Akshay Verma
>>
>> _______________________________________________
>> Python-nl mailing list
>> Python-nl at python.org
>> https://mail.python.org/mailman/listinfo/python-nl
>>
>>
>
> _______________________________________________
> Python-nl mailing list
> Python-nl at python.org
> https://mail.python.org/mailman/listinfo/python-nl
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-nl/attachments/20160904/0edb8b62/attachment-0001.html>