[pytest-dev] How to sandbox test execution with pytest

[Alessandro Amici]

Hi,

I still didn't find any ready-made tool to run pytest inside a docker
container so I ended up setting up my own solution.

I gave the gory details in the answer to my own question on StackOverflow:

http://stackoverflow.com/questions/35322452

but it all boils down to add a carefully crafted (read quite kludgy)
Dockerfile to the to folder of a project and run the container with a magic
set of options.

I hope you find it useful,
Alessandro



On Fri, 12 Feb 2016 at 09:06 Alessandro Amici <alexamici at gmail.com> wrote:

> Hi,
>
> while researching the use of pytest-xdist for my CI-like use case and I
> realised that having a "disposable" environment is as important as having
> a isolated one, so that every test or every session runs from the same
> initial state and it is not influenced by what older sessions might have
> left on folders writable by the *testuser* (/home/testuser, /tmp, /var/tmp,
> etc).
>
> This made me go back searching the web again for a way to use pytest with
> docker and I'm currently evaluating the following projects:
>
> https://github.com/search?q=pytest+docker
>
> Does anybody have experience with pytest/docker integration or with any of
> these projects?
>
> I'll report back what I find.
>
> Thanks,
> Alessandro
>
> On Thu, 11 Feb 2016 at 11:07 Alessandro Amici <alexamici at gmail.com> wrote:
>
>> Ionel and Holger,
>>
>> I had considered the --box option, but it only implements process-level
>> isolation which is not enough for my use case as I want to protect also
>> from unrestricted filesystem read/write access. Sorry, I should have
>> mentioned it in the question.
>>
>> Holger's suggestion instead looks almost good enough as user-level
>> isolation is strong and highly customizable. The only problem is that you
>> need administration access to the machine to create and configure a new
>> user. I'll look into it as I've never used the --tx option.
>>
>> Thanks,
>> Alessandro
>>
>>
>> On Thu, 11 Feb 2016 at 10:44 holger krekel <holger at merlinux.eu> wrote:
>>
>>> On Thu, Feb 11, 2016 at 11:39 +0200, Ionel Cristian Mărieș wrote:
>>> > There's the `--boxed` option (from
>>> https://pypi.python.org/pypi/pytest-xdist)
>>> > but you can easily do your own "insulator" by using fork syscall.
>>>
>>> Also one can use plain "xdist" which provides process-level isolation.
>>> It shouldn't be too hard to also use another unix user via e.g.:
>>>
>>>     py.test --tx ssh=OTHERUSER at localhost --dist=each
>>>
>>> but you need to have setup rsyncing, see
>>> https://pypi.python.org/pypi/pytest-xdist
>>> for some more info.  The current "remote" test running is a bit limited
>>> and we plan to improve xdist distribution during the upcoming pytest
>>> sprint,
>>> see
>>> https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/x/4034848#/
>>> creating an issue for "user-level separation of running tests" or
>>> something
>>> similar and adding a little donation is one way to help it :)
>>>
>>> best,
>>> holger
>>>
>>>
>>> >
>>> > Thanks,
>>> > -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>>> >
>>> > On Thu, Feb 11, 2016 at 10:33 AM, Alessandro Amici <
>>> alexamici at gmail.com>
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > I asked the following question on StackOverflow first
>>> > > http://stackoverflow.com/questions/35322452, but someone quickly
>>> (and
>>> > > wrongly in my opinion) marked that as duplicate of the general
>>> "sandboxing
>>> > > in python", so I can't get answers there anymore :(
>>> > >
>>> > > Here is the question:
>>> > >
>>> > > I'm interested in executing potentially untrusted tests with pytest
>>> in
>>> > > some kind of sandbox, similarly to what continuous integration
>>> services do.
>>> > >
>>> > > I understand that to properly sandbox a python process you need
>>> OS-level
>>> > > isolation, like running the tests in a disposable chroot/container,
>>> but in
>>> > > my use case I don't need to protect against intentionally malicious
>>> code,
>>> > > only from dangerous behaviour of pairing "randomly" functions with
>>> > > arguments. So lesser strict sandboxing may still be acceptable. But I
>>> > > didn't find any plugin that enables any form of sandboxing.
>>> > >
>>> > > What is the best way to sandbox tests execution in pytest?
>>> > >
>>> > > **Update**: This question is not about [python sandboxing in
>>> general](
>>> > >
>>> http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python
>>> )
>>> > > as the tests' code is run by pytest and I can't change the way it is
>>> > > executed to use `exec` or `ast` or whatever. Also using pypy-sandbox
>>> is not
>>> > > an option unfortunately as it is "a prototype only" as per the [PyPy
>>> > > feature page](http://pypy.org/features.html).
>>> > >
>>> > > End of question.
>>> > >
>>> > > For context I'm writing a pytest plugin that lets you search
>>> installed
>>> > > modules for functions that pass the given feature-specification tests
>>> > > (still alpha):
>>> > >
>>> > > https://github.com/alexamici/pytest-wish
>>> > >
>>> > > If you find the question useful and not answered already on
>>> stackoverflow,
>>> > > the best course of action would be to reopen it and answer there,
>>> but I'll
>>> > > take also an answer here :)
>>> > >
>>> > > Alessandro
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > pytest-dev mailing list
>>> > > pytest-dev at python.org
>>> > > https://mail.python.org/mailman/listinfo/pytest-dev
>>> > >
>>> > >
>>>
>>> > _______________________________________________
>>> > pytest-dev mailing list
>>> > pytest-dev at python.org
>>> > https://mail.python.org/mailman/listinfo/pytest-dev
>>>
>>>
>>> --
>>> about me:    http://holgerkrekel.net/about-me/
>>> contracting: http://merlinux.eu
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20160223/8f3010cd/attachment.html>