[pytest-dev] xdist and data sharing during execution in parallel

[Bruno Oliveira]

Hi Ayelet,

According to
http://codespeak.net/execnet/example/hybridpython.html#dumps-loads-examples,
execnet only supports built-in types, so trying to send
`node.config.global_data` to the slaves won't work.

I'm assuming that `PersistentDict` is similar to `shelve`, so another
option would be to try to construct a new instance of `PersistentDict` on
each slave that share the same underlying file.

Also can you provide some examples on how do you plan to use that shared
`PersistentDict` in each slave?

Cheers,


On Wed, Sep 17, 2014 at 1:32 PM, Ayelet Regev <ayelet at fabrixsystems.com>
wrote:

>  Hi,
>
> I'm creating a time-line based on pytest.
> im using xdist with multiple slaves (in the amount of the tests). Each
> test is started by time and dependency. which mean some of the gateways
> will be waiting for others to be "Done".
>
> for that, io need some data sharing between them all, and cannot use
> "slaveoutput".
>
> i created my own dict object using file handler, and i want to create
> instance once on master and share it will all slaves.
>
> but i cannot pass object to "slaveinput"
>
> Do you have any advice?
>
>
>
> *def pytest_configure(config):*
>
>     config.global_time_line_json = config.getoption("timeline",None)
>     unicode_to_ascii(config.global_time_line_json)
>     if config.global_time_line_json is None:
>         raise Exception("no timeline data was provided to command")
>     config.properties = config.getoption("properties")
>     unicode_to_ascii(config.properties)
>     logfile = config.properties.get("logfile",None)
>     if logfile is None:
>         raise Exception("no logfile path was provided to command under
> properties")
>     config.flow_dir = config.properties.get("flow_dir",None)
>     logfile_basename = os.path.basename(logfile)
>     _logger = Logger(config.flow_dir,logfile_basename)
>     config.logger = _logger.logger()
>     config.active_dir = "%s/%s" % (config.flow_dir,ACTIVE_DIR)
>     *if not hasattr(config,"slaveinput"):*
> *        config.logger.info2("Master")*
> *        config.global_data = PersistentDict('%s/%s' %
> (config.active_dir,FLOW_JSON), 'c', format='json')*
>
> *def pytest_configure_node(node):*
>     # runs only on Master
>     node.config.logger.info2("on Master sending data to Slave")
>    * node.slaveinput["global_data"] = node.config.global_data*
>
> --
> Best Regards,
>
> Ayelet Regev Dabah
> Email: ayelet at fabrixsystems.com
>
>
> _______________________________________________
> Pytest-dev mailing list
> Pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20140924/b2083ff8/attachment.html>