Hey, Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well. This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really. Mark
On 08/24/2012 08:40 PM, mark florisson wrote:
Hey,
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well.
This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs
So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really.
I think I support using a git submodule for now. Not sure about making it optional (which I assume would make array expression testcases not run if it minivect is not present so that there's no test failures); we want to make sure we are forced to do releases and testing right and include it, and if it is only required to compile code that uses memoryview expressions users could get confused about there being two Cython "editions" around. Since "git submodule" does link to a specific revision, so there's no stability concerns over verbatim inclusion. How hg-git deals with submodules is worth consideration too though. Dag
On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote:
On 08/24/2012 08:40 PM, mark florisson wrote:
Hey,
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well.
This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs
So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really.
I think I support using a git submodule for now. Not sure about making it optional (which I assume would make array expression testcases not run if it minivect is not present so that there's no test failures); we want to make sure we are forced to do releases and testing right and include it, and if it is only required to compile code that uses memoryview expressions users could get confused about there being two Cython "editions" around.
Since "git submodule" does link to a specific revision, so there's no stability concerns over verbatim inclusion.
How hg-git deals with submodules is worth consideration too though.
Another option you didn't mention is to push the responsibility of getting minivect over to end-users; Cython simply tries to do "import minivect". This does have versioning issues though since git will likely not be the tool used to fetch the revision. A lot more pain for those who uses array expressions, but a little less pain for the rest. So it depends on how you weigh the user groups. Realistically, we'd want to depend on LLVM as well down the road for minivect stuff (at least if you want optimal performance), so perhaps opening the can-of-external-dependency-worms should just be done sooner rather than later. Dag
Dag Sverre Seljebotn, 27.08.2012 11:55:
On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote:
On 08/24/2012 08:40 PM, mark florisson wrote:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well.
This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs
So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really.
I think I support using a git submodule for now. Not sure about making it optional (which I assume would make array expression testcases not run if it minivect is not present so that there's no test failures); we want to make sure we are forced to do releases and testing right and include it, and if it is only required to compile code that uses memoryview expressions users could get confused about there being two Cython "editions" around.
Since "git submodule" does link to a specific revision, so there's no stability concerns over verbatim inclusion.
How hg-git deals with submodules is worth consideration too though.
Another option you didn't mention is to push the responsibility of getting minivect over to end-users; Cython simply tries to do "import minivect". This does have versioning issues though since git will likely not be the tool used to fetch the revision.
A lot more pain for those who uses array expressions, but a little less pain for the rest. So it depends on how you weigh the user groups.
Realistically, we'd want to depend on LLVM as well down the road for minivect stuff (at least if you want optimal performance), so perhaps opening the can-of-external-dependency-worms should just be done sooner rather than later.
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt. As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while. For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency). Stefan
On 27 August 2012 12:06, Stefan Behnel <stefan_ml@behnel.de> wrote:
Dag Sverre Seljebotn, 27.08.2012 11:55:
On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote:
On 08/24/2012 08:40 PM, mark florisson wrote:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well.
This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs
So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really.
I think I support using a git submodule for now. Not sure about making it optional (which I assume would make array expression testcases not run if it minivect is not present so that there's no test failures); we want to make sure we are forced to do releases and testing right and include it, and if it is only required to compile code that uses memoryview expressions users could get confused about there being two Cython "editions" around.
Since "git submodule" does link to a specific revision, so there's no stability concerns over verbatim inclusion.
How hg-git deals with submodules is worth consideration too though.
Another option you didn't mention is to push the responsibility of getting minivect over to end-users; Cython simply tries to do "import minivect". This does have versioning issues though since git will likely not be the tool used to fetch the revision.
A lot more pain for those who uses array expressions, but a little less pain for the rest. So it depends on how you weigh the user groups.
Realistically, we'd want to depend on LLVM as well down the road for minivect stuff (at least if you want optimal performance), so perhaps opening the can-of-external-dependency-worms should just be done sooner rather than later.
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita). I'll make minivect distributable, then.
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote:
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project. The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable". BTW, have you been in touch with the PyPy people about this? They should have implemented their own vectorisation for what they call numpypy. Maybe your tool would be of interest for them as well? Stefan
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote:
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed. E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
BTW, have you been in touch with the PyPy people about this? They should have implemented their own vectorisation for what they call numpypy. Maybe your tool would be of interest for them as well?
I was thinking about it, not yet. I think it'd have to be adapted somehow, since it uses llvmpy. It's also not entirely featureful yet (e.g. no reductions), since most of the focus has been on performance. But I'll raise the issue on the ML, thanks :)
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote:
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there? Stefan
On 08/27/2012 02:05 PM, Stefan Behnel wrote:
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote:
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there?
Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all. Dag
On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote:
On 08/27/2012 02:05 PM, Stefan Behnel wrote:
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote:
My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt.
As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while.
For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there?
Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all.
I think there's a case for having minivect as a git submodule for convenient development purposes (so that you can easily sort of refactor Cython and minivect together), but that's only a development mechanism. I.e., Cython first tries to import 'Cython.minivect' (which could be a symlink into a git submodule) and if that fails 'minivect', the latter would be the normal end-user mode because Cython doesn't bundle minivect. Dag
Dag Sverre Seljebotn, 27.08.2012 14:47:
On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote:
On 08/27/2012 02:05 PM, Stefan Behnel wrote:
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel wrote:
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there?
Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all.
I think there's a case for having minivect as a git submodule for convenient development purposes (so that you can easily sort of refactor Cython and minivect together), but that's only a development mechanism.
I.e., Cython first tries to import 'Cython.minivect' (which could be a symlink into a git submodule) and if that fails 'minivect', the latter would be the normal end-user mode because Cython doesn't bundle minivect.
python setup.py develop (or the equivalent options for pip and easy_install) Stefan
On Mon, Aug 27, 2012 at 5:47 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote:
On 08/27/2012 02:05 PM, Stefan Behnel wrote:
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13:
On 27 August 2012 12:06, Stefan Behnel wrote: > > My experience with lxml tells me that it's often better to keep > things > separate but integrated, instead of shipping them in a big box. As > long as > it doesn't hurt too much to have separate tools, we should keep it > that > way. Those who prefer everything in a big box can use > distributions like > Sage, or use apt. > > As for versioning, you can set dependency version ranges in > distutils (and > friends) which are honoured by install tools like pip. That keeps the > installation fully automatic (and definitely not "a lot more > pain"). That > being said, the best way to handle this is to build a well defined > interface between the two components and to keep that alive for a > while. > > For Jenkins, we'd set up separate jobs that build the dependencies > and then > install them from there before running the integration tests. We > could even > have dedicated integration test jobs that only run the tests that > involve > the dependency (and potentially more than one version of the > dependency).
Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there?
Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all.
I think there's a case for having minivect as a git submodule for convenient development purposes (so that you can easily sort of refactor Cython and minivect together), but that's only a development mechanism.
I.e., Cython first tries to import 'Cython.minivect' (which could be a symlink into a git submodule) and if that fails 'minivect', the latter would be the normal end-user mode because Cython doesn't bundle minivect.
+1, this seems like the best option to me. - Robert
On 08/27/2012 08:07 PM, Robert Bradshaw wrote:
On Mon, Aug 27, 2012 at 5:47 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote:
On 08/27/2012 02:05 PM, Stefan Behnel wrote:
mark florisson, 27.08.2012 13:52:
On 27 August 2012 12:40, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 27.08.2012 13:13: > > On 27 August 2012 12:06, Stefan Behnel wrote: >> >> My experience with lxml tells me that it's often better to keep >> things >> separate but integrated, instead of shipping them in a big box. As >> long as >> it doesn't hurt too much to have separate tools, we should keep it >> that >> way. Those who prefer everything in a big box can use >> distributions like >> Sage, or use apt. >> >> As for versioning, you can set dependency version ranges in >> distutils (and >> friends) which are honoured by install tools like pip. That keeps the >> installation fully automatic (and definitely not "a lot more >> pain"). That >> being said, the best way to handle this is to build a well defined >> interface between the two components and to keep that alive for a >> while. >> >> For Jenkins, we'd set up separate jobs that build the dependencies >> and then >> install them from there before running the integration tests. We >> could even >> have dedicated integration test jobs that only run the tests that >> involve >> the dependency (and potentially more than one version of the >> dependency). > > > Oh great, that works for me as well. I thought Cython's take was to > avoid dependencies by shipping everything (like Plex and Tempita).
Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped.
> I'll make minivect distributable, then.
I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project.
The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable".
Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed.
I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup.
E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project.
Doesn't that apply to more or less all code out there?
Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all.
I think there's a case for having minivect as a git submodule for convenient development purposes (so that you can easily sort of refactor Cython and minivect together), but that's only a development mechanism.
I.e., Cython first tries to import 'Cython.minivect' (which could be a symlink into a git submodule) and if that fails 'minivect', the latter would be the normal end-user mode because Cython doesn't bundle minivect.
+1, this seems like the best option to me.
I'm not convinced by "setup.py develop" either (I tend to keep distutils out of my workflow whenever I can). Dag
While I have not tried this yet, array expressions in Cython might be the final nails in the coffin for Fortran 90 as far as I am concerned. Great work! :-) Sturla On 24.08.2012 20:40, mark florisson wrote:
Hey,
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well.
This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs
So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really.
Mark _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? Stefan
On 12 October 2012 08:36, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation.
Did you manage to split off a separate minivect package?
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
I have some time this weekend, I'll see if I can finish it then. Shall I make a new PR or merge it directly?
mark florisson, 12.10.2012 12:14:
On 12 October 2012 08:36, Stefan Behnel wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation.
Did you manage to split off a separate minivect package?
I have some time this weekend, I'll see if I can finish it then. Shall I make a new PR or merge it directly?
As I said, I won't review it anyway, so it's no difference for me. But given that your original pull request didn't receive any comments, I doubt that making a new one would serve a major purpose. Stefan
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 12 October 2012 08:36, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation.
Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). I don't see a reason for a new pull request. - Robert
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 12 October 2012 08:36, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation.
Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. Dag Sverre
I don't see a reason for a new pull request.
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On 12 October 2012 20:01, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 12 October 2012 08:36, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation.
Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
Dag Sverre
I don't see a reason for a new pull request.
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, and changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
On 12 October 2012 08:36, Stefan Behnel wrote:
mark florisson, 24.08.2012 20:40:
Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, and changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. Stefan
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
On 12 October 2012 08:36, Stefan Behnel wrote:
mark florisson, 24.08.2012 20:40: > Here a pull request for element-wise array expressions for Cython: > https://github.com/cython/cython/pull/144
Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) Dag Sverre
On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
On 12 October 2012 08:36, Stefan Behnel wrote: > mark florisson, 24.08.2012 20:40: >> Here a pull request for element-wise array expressions for Cython: >> https://github.com/cython/cython/pull/144 > > Mark, any news on this? I'd like to see a version merged before > the master branch starts diverging all too far - it already > requires a bit of adaptation. > Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Also, I don't even know when/if numba will use it... Mark, are you able to fill us in on that? Dag Sverre
On 14 October 2012 09:23, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: > > On 12 October 2012 08:36, Stefan Behnel wrote: >> >> mark florisson, 24.08.2012 20:40: >>> >>> Here a pull request for element-wise array expressions for Cython: >>> https://github.com/cython/cython/pull/144 >> >> >> Mark, any news on this? I'd like to see a version merged before >> the master branch starts diverging all too far - it already >> requires a bit of adaptation. >> Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Also, I don't even know when/if numba will use it... Mark, are you able to fill us in on that?
Yes, numba uses minivect, currently as a subtree (did that yesterday). Submodules are very bad, especially if you develop the submodule itself, and switch branches in the superproject (it will just forget about unpinned changes, which means you just lose your work without so much as a warning). Numba mostly uses and extends the minivect type system and related type conversions (to/from numpy dtypes, to/from ctypes, etc). NumbaPro uses it for array expressions and ufuncs, which will at some point make it into numba. You're entirely right in saying that it breathes only through other projects. I'll add the rest of the response in answer to Stefan's question.
Dag Sverre _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
On 12 October 2012 08:36, Stefan Behnel wrote: > mark florisson, 24.08.2012 20:40: >> Here a pull request for element-wise array expressions for Cython: >> https://github.com/cython/cython/pull/144 > > Mark, any news on this? I'd like to see a version merged before > the master branch starts diverging all too far - it already > requires a bit of adaptation. > Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages? Stefan
On 10/14/2012 10:32 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: > On 12 October 2012 08:36, Stefan Behnel wrote: >> mark florisson, 24.08.2012 20:40: >>> Here a pull request for element-wise array expressions for Cython: >>> https://github.com/cython/cython/pull/144 >> >> Mark, any news on this? I'd like to see a version merged before >> the master branch starts diverging all too far - it already >> requires a bit of adaptation. >> Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages?
Mark already raised the issue and we discussed in through in another thread where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now. If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here. (Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up. Mark had a weekend of spare time to get this merged (yay!), and now who-knows-what happens because you decided to bikeshed something that had, at least to the eyes of rest of us, achieved consensus on this list.) Dag
Dag Sverre Seljebotn, 14.10.2012 11:03:
On 10/14/2012 10:32 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote: > On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >> On 12 October 2012 08:36, Stefan Behnel wrote: >>> mark florisson, 24.08.2012 20:40: >>>> Here a pull request for element-wise array expressions for Cython: >>>> https://github.com/cython/cython/pull/144 >>> >>> Mark, any news on this? I'd like to see a version merged before >>> the master branch starts diverging all too far - it already >>> requires a bit of adaptation. >>> Did you manage to split off a separate minivect package? > > I'm assuming this has already been looked at, at least to some level, > by Dag, but I'll try to take a brief pass at it too (probably more the > interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
> I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages?
Mark already raised the issue and we discussed in through in another thread
This is still the same thread, at least for me. I replied to it when asking for progress.
where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now.
If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here.
The single argument for git based integration was that it simplifies the workflow during development. Meaning, it's a development-only thing and deployments would use separate package installations. I think that's ok as long as it's actually helpful for development and doesn't start getting in the way. I can't comment on that yet, but I wouldn't want to have to deal with it.
(Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up.
It may be a trivial detail as long as it only has a reasonable impact on the development. If it has a user impact, it's no longer immediately obvious that it's trivial.
Mark had a weekend of spare time to get this merged (yay!), and now who-knows-what happens because you decided to bikeshed something that had, at least to the eyes of rest of us, achieved consensus on this list.)
Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of what he's doing now just wasn't clear enough to me. Mark, could you explain in a bit more detail in which cases the git version will be used and in which cases users would (or can) install minivect separately? Stefan
On 10/14/2012 12:23 PM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 14.10.2012 11:03:
On 10/14/2012 10:32 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: > On 10/12/2012 05:50 PM, Robert Bradshaw wrote: >> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >>> On 12 October 2012 08:36, Stefan Behnel wrote: >>>> mark florisson, 24.08.2012 20:40: >>>>> Here a pull request for element-wise array expressions for Cython: >>>>> https://github.com/cython/cython/pull/144 >>>> >>>> Mark, any news on this? I'd like to see a version merged before >>>> the master branch starts diverging all too far - it already >>>> requires a bit of adaptation. >>>> Did you manage to split off a separate minivect package? >> >> I'm assuming this has already been looked at, at least to some level, >> by Dag, but I'll try to take a brief pass at it too (probably more the >> interface than the implementation). > > Thanks for doing that, it'd be great to get this in (but myself I've got > nothing to spare). I'll admit I was mostly focused on the generated > code and > the algorithms in minivect rather than the integration with Cython. > >> I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages?
Mark already raised the issue and we discussed in through in another thread
This is still the same thread, at least for me. I replied to it when asking for progress.
where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now.
If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here.
The single argument for git based integration was that it simplifies the workflow during development. Meaning, it's a development-only thing and deployments would use separate package installations.
I think that's ok as long as it's actually helpful for development and doesn't start getting in the way. I can't comment on that yet, but I wouldn't want to have to deal with it.
You know what -- I didn't think things through and was totally wrong here. 'git subtree' has a very different feel to it than 'git submodule'; it means that minivect will be in the tree by default and would have to be stripped out when making a release (or included by default). So revisiting this discussion is indeed in order, as git subtree wasn't mentioned last time. Dag Sverre
(Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up.
It may be a trivial detail as long as it only has a reasonable impact on the development. If it has a user impact, it's no longer immediately obvious that it's trivial.
Mark had a weekend of spare time to get this merged (yay!), and now who-knows-what happens because you decided to bikeshed something that had, at least to the eyes of rest of us, achieved consensus on this list.)
Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of what he's doing now just wasn't clear enough to me.
Mark, could you explain in a bit more detail in which cases the git version will be used and in which cases users would (or can) install minivect separately?
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On 14 October 2012 11:23, Stefan Behnel <stefan_ml@behnel.de> wrote:
Dag Sverre Seljebotn, 14.10.2012 11:03:
On 10/14/2012 10:32 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: > On 10/12/2012 05:50 PM, Robert Bradshaw wrote: >> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >>> On 12 October 2012 08:36, Stefan Behnel wrote: >>>> mark florisson, 24.08.2012 20:40: >>>>> Here a pull request for element-wise array expressions for Cython: >>>>> https://github.com/cython/cython/pull/144 >>>> >>>> Mark, any news on this? I'd like to see a version merged before >>>> the master branch starts diverging all too far - it already >>>> requires a bit of adaptation. >>>> Did you manage to split off a separate minivect package? >> >> I'm assuming this has already been looked at, at least to some level, >> by Dag, but I'll try to take a brief pass at it too (probably more the >> interface than the implementation). > > Thanks for doing that, it'd be great to get this in (but myself I've got > nothing to spare). I'll admit I was mostly focused on the generated > code and > the algorithms in minivect rather than the integration with Cython. > >> I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages?
Mark already raised the issue and we discussed in through in another thread
This is still the same thread, at least for me. I replied to it when asking for progress.
where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now.
If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here.
The single argument for git based integration was that it simplifies the workflow during development. Meaning, it's a development-only thing and deployments would use separate package installations.
I think that's ok as long as it's actually helpful for development and doesn't start getting in the way. I can't comment on that yet, but I wouldn't want to have to deal with it.
(Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up.
It may be a trivial detail as long as it only has a reasonable impact on the development. If it has a user impact, it's no longer immediately obvious that it's trivial.
Mark had a weekend of spare time to get this merged (yay!), and now who-knows-what happens because you decided to bikeshed something that had, at least to the eyes of rest of us, achieved consensus on this list.)
Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of what he's doing now just wasn't clear enough to me.
Mark, could you explain in a bit more detail in which cases the git version will be used and in which cases users would (or can) install minivect separately?
The subtree plan is to always use the included version, which is also automatic for a git checkout. The dependency plan is to support pip/easy_install, pinned on a specific minivect release. Another problem with B) arises when minivect is included in other projects as a dependency, and the projects have conflicting minivect version requirements.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On 14 October 2012 09:32, Stefan Behnel <stefan_ml@behnel.de> wrote:
Dag Sverre Seljebotn, 14.10.2012 10:18:
On 10/14/2012 08:18 AM, Stefan Behnel wrote:
mark florisson, 13.10.2012 20:30:
On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: > On 12 October 2012 08:36, Stefan Behnel wrote: >> mark florisson, 24.08.2012 20:40: >>> Here a pull request for element-wise array expressions for Cython: >>> https://github.com/cython/cython/pull/144 >> >> Mark, any news on this? I'd like to see a version merged before >> the master branch starts diverging all too far - it already >> requires a bit of adaptation. >> Did you manage to split off a separate minivect package?
I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation).
Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython.
I don't see a reason for a new pull request.
Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories.
This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history).
What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users.
I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it.
This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level.
Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is).
Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time.
I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless.
I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.)
Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages?
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point. For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4. Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba. We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately. As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway. Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases. BTW, I would guess that no-one has tested hg with git subtrees yet? Stefan
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history. I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
BTW, I would guess that no-one has tested hg with git subtrees yet?
Heh, no. The hg support didn't work great for me (it was really slow), can I ask why you're using it?
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
mark florisson, 14.10.2012 15:13:
The hg support didn't work great for me (it was really slow),
It's fast enough for almost everything. What's really slow is when it has to figure out how to match branches after head changes - I consider that a pretty serious bug. But that happens quite rarely. And I usually push only the master branch anyway, which is quick. Obviously, hg is way faster against a mercurial server than against a git server.
can I ask why you're using it?
I prefer the interface of hg over that of git, that's the main reason. In fact, whenever I have to touch git, it quickly lives up to its name: http://dict.leo.org/ende?lang=en&search=igitt Stefan
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system). - Robert
On 16 October 2012 18:48, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system).
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
from Cython.Compiler import Main [98632 refs] ^D [98632 refs]
Great, thanks for the review Robert. There seems to be a refcount issue with python 3: cython@sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$ /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. The object being visited is a UtilityCode object. I think the error means the object is being visited more often than it's refcount value, which means it's not increffed properly somewhere. I'm not sure how/if my changes introduced this, has this problem been encountered recently in Cython's master?
On 21 October 2012 13:39, mark florisson <markflorisson88@gmail.com> wrote:
On 16 October 2012 18:48, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system).
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Great, thanks for the review Robert. There seems to be a refcount issue with python 3:
from Cython.Compiler import Main [98632 refs] ^D [98632 refs]
cython@sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$ /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
The object being visited is a UtilityCode object. I think the error means the object is being visited more often than it's refcount value, which means it's not increffed properly somewhere. I'm not sure how/if my changes introduced this, has this problem been encountered recently in Cython's master?
This is running with -m trace --trace: <snip> Builtin.py(396): for bf in builtin_function_table: Builtin.py(397): bf.declare_in_scope(builtin_scope) --- modulename: Builtin, funcname: declare_in_scope Builtin.py(163): func_type, sig = self.func_type, self.sig Builtin.py(164): if func_type is None: Builtin.py(165): if sig is None: Builtin.py(166): sig = Signature(self.args, self.ret_type) --- modulename: TypeSlots, funcname: __init__ TypeSlots.py(85): self.has_dummy_arg = 0 TypeSlots.py(86): self.has_generic_args = 0 TypeSlots.py(87): if arg_format[:1] == '-': TypeSlots.py(90): if arg_format[-1:] == '*': TypeSlots.py(93): self.fixed_arg_format = arg_format TypeSlots.py(94): self.ret_format = ret_format TypeSlots.py(95): self.error_value = self.error_value_map.get(ret_format, None) TypeSlots.py(96): self.is_staticmethod = False Builtin.py(167): func_type = sig.function_type() --- modulename: TypeSlots, funcname: function_type TypeSlots.py(125): args = [] TypeSlots.py(126): for i in range(self.num_fixed_args()): --- modulename: TypeSlots, funcname: num_fixed_args TypeSlots.py(99): return len(self.fixed_arg_format) TypeSlots.py(127): if self_arg_override is not None and self.is_self_arg(i): TypeSlots.py(131): arg_type = self.fixed_arg_type(i) --- modulename: TypeSlots, funcname: fixed_arg_type TypeSlots.py(110): return self.format_map[self.fixed_arg_format[i]] TypeSlots.py(132): args.append(PyrexTypes.CFuncTypeArg("", arg_type, None)) --- modulename: PyrexTypes, funcname: __init__ PyrexTypes.py(2966): self.name = name PyrexTypes.py(2967): if cname is not None: PyrexTypes.py(2970): self.cname = Naming.var_prefix + name PyrexTypes.py(2971): self.type = type PyrexTypes.py(2972): self.pos = pos PyrexTypes.py(2973): self.needs_type_test = False # TODO: should these defaults be set in analyse_types()? TypeSlots.py(126): for i in range(self.num_fixed_args()): TypeSlots.py(127): if self_arg_override is not None and self.is_self_arg(i): TypeSlots.py(131): arg_type = self.fixed_arg_type(i) --- modulename: TypeSlots, funcname: fixed_arg_type TypeSlots.py(110): return self.format_map[self.fixed_arg_format[i]] TypeSlots.py(132): args.append(PyrexTypes.CFuncTypeArg("", arg_type, None)) --- modulename: PyrexTypes, funcname: __init__ PyrexTypes.py(2966): self.name = name PyrexTypes.py(2967): if cname is not None: PyrexTypes.py(2970): self.cname = Naming.var_prefix + name PyrexTypes.py(2971): self.type = type PyrexTypes.py(2972): self.pos = pos PyrexTypes.py(2973): self.needs_type_test = False # TODO: should these defaults be set in analyse_types()? TypeSlots.py(126): for i in range(self.num_fixed_args()): TypeSlots.py(133): if self_arg_override is not None and self.returns_self_type(): TypeSlots.py(136): ret_type = self.return_type() --- modulename: TypeSlots, funcname: return_type TypeSlots.py(113): return self.format_map[self.ret_format] TypeSlots.py(137): exc_value = self.exception_value() --- modulename: TypeSlots, funcname: exception_value TypeSlots.py(121): return self.error_value_map.get(self.ret_format) TypeSlots.py(138): return PyrexTypes.CFuncType(ret_type, args, exception_value = exc_value) --- modulename: PyrexTypes, funcname: __init__ PyrexTypes.py(2531): self.return_type = return_type python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. Aborted Maybe I'll try rebasing on the latest master again.
On Sun, Oct 21, 2012 at 5:39 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 16 October 2012 18:48, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system).
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Great, thanks for the review Robert. There seems to be a refcount issue with python 3:
from Cython.Compiler import Main [98632 refs] ^D [98632 refs]
cython@sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$ /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
The object being visited is a UtilityCode object. I think the error means the object is being visited more often than it's refcount value, which means it's not increffed properly somewhere. I'm not sure how/if my changes introduced this, has this problem been encountered recently in Cython's master?
Not that I'm aware of. The refnanny is supposed to catch this kind of thing, is it enabled? - Robert
On 23 October 2012 02:44, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 21, 2012 at 5:39 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 16 October 2012 18:48, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system).
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Great, thanks for the review Robert. There seems to be a refcount issue with python 3:
from Cython.Compiler import Main [98632 refs] ^D [98632 refs]
cython@sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$ /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
The object being visited is a UtilityCode object. I think the error means the object is being visited more often than it's refcount value, which means it's not increffed properly somewhere. I'm not sure how/if my changes introduced this, has this problem been encountered recently in Cython's master?
Not that I'm aware of. The refnanny is supposed to catch this kind of thing, is it enabled?
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Yeah, it doesn't catch it. I'll try tracing the imports to see what it is importing that is different from master. Thanks.
On 23 October 2012 10:47, mark florisson <markflorisson88@gmail.com> wrote:
On 23 October 2012 02:44, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 21, 2012 at 5:39 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 16 October 2012 18:48, Robert Bradshaw <robertwb@gmail.com> wrote:
On Sun, Oct 14, 2012 at 6:13 AM, mark florisson <markflorisson88@gmail.com> wrote:
On 14 October 2012 14:05, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 14.10.2012 13:59: > The problem with minivect as a package is that it caters to different > projects, which have different requirements. Cython and minivect are > quite closely coupled, and any future change, or in the future any > older version may not have the functionality Cython needs, it's not > exactly a stable API at this point.
Ok, understood.
> For instance Numba needs python > 2.7, whereas Cython needs to be compatible with python 2.4. > > Before releasing minivect I'll verify every time that it doesn't break > Cython, but I currently have no real promises for backwards or forward > compatibility. And that is really because not all use cases have yet > been anticipated, and some really require a change, as I've already > seen with Numba. > > We could list minivect as a dependency, which works for > easy_install/pip users, but I just foresee numerous people running > into problems that didn't install with pip, and I don't think an > exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Yes, definitely.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history.
I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree.
As I mentioned on the pull request, looks good to me. Given the (somewhat) tight coupling with the AST but the desire to use the codebase for multiple projects, a subtree seems to make the most sense (until/if we have some kind of a plugin system).
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Great, thanks for the review Robert. There seems to be a refcount issue with python 3:
from Cython.Compiler import Main [98632 refs] ^D [98632 refs]
cython@sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$ /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
The object being visited is a UtilityCode object. I think the error means the object is being visited more often than it's refcount value, which means it's not increffed properly somewhere. I'm not sure how/if my changes introduced this, has this problem been encountered recently in Cython's master?
Not that I'm aware of. The refnanny is supposed to catch this kind of thing, is it enabled?
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Yeah, it doesn't catch it. I'll try tracing the imports to see what it is importing that is different from master. Thanks.
I had some time to look into it, and it appeared that the bug was in CloneNode, it just never got triggered, unless you analyse the types of the CloneNode. That sets is_temp to True, but the node that's being cloned may not own its reference, but temporaries are assumed to own it.
On 10/14/2012 03:05 PM, Stefan Behnel wrote:
mark florisson, 14.10.2012 13:59:
The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point.
Ok, understood.
For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4.
Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba.
We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that.
Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately.
As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway.
Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases.
+1 to Mark's proposal, this seems nice.
BTW, I would guess that no-one has tested hg with git subtrees yet?
git subtree is just a utility for pushing/pulling the history of a subdirectory to/from another repository. As long as you don't need to update to a new version of minivect, you shouldn't be affected at all; when you don't actually use the "git subtree" commands, it's just a normal directory under git control. But if one needs to fix a bug in minivect when doing Cython development, I'm guessing hg-git can't help you move the patch upstream (but you can always just submit a normal Unix diff upstream instead if you don't want to use git). Dag Sverre
participants (5)
-
Dag Sverre Seljebotn -
mark florisson -
Robert Bradshaw -
Stefan Behnel -
Sturla Molden