Mailman 3 - Python.NET

Working group notes for November 21, 2019
by Mark Visser 21 Nov '19

21 Nov '19

Attendees: Victor, Benoit, Mark, Amos, Benedikt Agenda Review action items from last meeting: Mark - got in touch with .NET foundation Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Victor - will create Github issues for items in the Quantconnect PR and figure out plan for bringing more generic solutions to master. ? - can someone research how hard it would be to move the CI to Github Actions? Benoit - will break out SlotsHolder and __import__ restore into smaller PRs Amos - will continue to break https://github.com/pythonnet/pythonnet/pull/958 <https://github.com/pythonnet/pythonnet/pull/958> into smaller PRs following Benoit's example All - please discuss changing the meeting time on the list Notes .NET foundation - Benedikt working on it with contact & maintainer of IronPython Multithreaded python code - Benoit still working on it, has a deadlock issue that's probably unrelated. WPF has a similar issue? There are old tickets around where people tried this before. Possibility of making it automatic, but requires a major version bump. https://github.com/pythonnet/pythonnet/issues/990 <https://github.com/pythonnet/pythonnet/pull/958> for example. Quantconnect PR - perf PR was OK, but is a fairly small part of the larger PR. Need to get .NET standard / core stuff on the way so that they don't fork off. Github actions - currently have appveyer and travis CI implementations, but in theory github actions can do both and also work on mac. Could cover all platforms. One of the unit tests crashes on Mac - ongoing problem, would be good to have CI for that. Possible plan is to start with github actions on Mac, then extend to Linux, finally Windows. __import__ restore - merged (moved out of pull 958) SlotsHolder in progress - goal is to make it so that PyFinalize doesn't need to be called to disconnect C# from Python (moved out of pull 958) General note: please make PRs as small and targeted as possible - large PRs are very difficult to review and judge the impact and risk. Large PRs won't likely be merged. Discussion around time to accommodate Amos - we will move the meeting 3 hours earlier. Need to start planning for a point release (2.4.1) including Python 3.8 support. PyBuffer change - good addition but is crashing in a non-obvious way. Maybe not a candidate for the point release. If someone needs it for the next release, they could try to fix it in branch. We shouldn't have any breaking changes since 2.4.0, only bug fixes. Benedikt - Haven't been able to integrate CLR loader yet (ripping out old stuff). Will try to have the interface ready for the CLR loader. Python.NET exposes a clr module inside Python which exposes .NET VM startup functions. Mono vs. Core have different code paths currently. Current approach in latest .NET core seems to be stable. New CLR loader abstracts three different ways of loading .NET. Technical discussion. Proposal: pass parameters as JSON. Alternatives? Would also enable choosing runtime version of Python. Benchmarking - Still in progress. PR will be open today. Could CLR loader be completely separate from Python.NET? Benedikt - sure, it's useful on its own as a low-level mechanism to talk to .NET. On the other hand Python interop is interconnected with how Python.NET works and should stay in the project. No plans to have more libraries than CLR loader (which isn't actually needed if you're embedding Python within .NET) and pure interop + abstractions on top. Action Items Benedikt - continue working on joining .NET foundation Amos - continue breaking up pull 958 into smaller PRs to make it easier to review and merge. e.g. have a single PR that gets reference counts right and doesn't touch GC chains. See Benoit's example of the __import__ restore PR for best practices. Amos - same for other older PRs Mark - move meeting 3 hours earlier Benedikt - will try to reproduce PyBuffer crash locally Benedikt - will check if there are any breaking changes since 2.4.0 Benedikt - continue exploring CLR loader, splitting parts into separate Python interop DLLs, have main Python runtime implement layers on top The meeting notes google doc is here <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…>. Feel free to correct or add additional information. The next working group meeting will be held on December 5th at 12pm EST, 9am PST, 6pm CET, 1am China. Please note that this is three hours earlier than previous meetings. Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Working group meeting notes for November 7, 2019
by Mark Visser 19 Nov '19

19 Nov '19

Attendees: Benoit Hudson (ImgSpc), Amos Li, Mark Visser (Unity) Agenda Review action items from last meeting Notes Update from Victor: The only thing I’ve done though is reviewing PyBuffer request PR: https://github.com/pythonnet/pythonnet/pull/980 <https://github.com/pythonnet/pythonnet/pull/980>, for which, sadly, tests are still not passing. Update from Benedikt: I've been mostly working on clr_loader, will open a PR on pythonnet for this weekend. .NET foundation: Mark finally got a reply back from his contact at Unity He will loop Benedikt in on the e-mail as soon as he's introduced to someone on the .NET foundation Soft restart: PR is large and hard to review https://github.com/pythonnet/pythonnet/pull/958#issuecomment-551230878 <https://github.com/pythonnet/pythonnet/pull/958#issuecomment-551230878> documented here: https://docs.google.com/document/d/1a9OLsdKHXJ6MxHjo0WlVRcfjAiaP5hONNKUKKH6… <https://docs.google.com/document/d/1a9OLsdKHXJ6MxHjo0WlVRcfjAiaP5hONNKUKKH6…> Benoit will break out SlotsHolder and __import__ restore into smaller PRs Amos can you continue to break https://github.com/pythonnet/pythonnet/pull/958 <https://github.com/pythonnet/pythonnet/pull/958> into smaller PRs to land individually? discussion about system modules restoration - clarity found split removal of native code page into another PR after everything else has landed Regarding the meeting time: Amos pointed out that the meeting is at 4am his local time - could we move this meeting to a better time for him? possible times: 4 hours earlier (12pm EST, 8am PST, 12am China) 2 hours later (5pm EST, 11am PST, 6am China) 3 hours later (6pm EST, 12pm PST, 7am China) please discuss! Action Items Mark - still working on getting in touch with .NET foundation Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Victor - will create Github issues for items in the Quantconnect PR and figure out plan for bringing more generic solutions to master. ? - can someone research how hard it would be to move the CI to Github Actions? Benoit - will break out SlotsHolder and __import__ restore into smaller PRs Amos - will continue to break https://github.com/pythonnet/pythonnet/pull/958 <https://github.com/pythonnet/pythonnet/pull/958> into smaller PRs following Benoit's example All - please discuss changing the meeting time on the list

2 3

Working group minutes for October 22, 2019
by Mark Visser 22 Oct '19

22 Oct '19

Attendees: Benedikt Reinhartz (Project Lead), Benoit Hudson (ImgSpc), Felix Bourbonnais (ImgSpc), Mohamed Kouba (ANSYS Inc.), Victor Milovanov, Amos Li, Mark Visser (Unity) Agenda (Mark) I got a reply from Jared Broad (QuantConnect): His team is unable to devote resources to anything other than the PR they agreed to open (#895). He wanted to remind us that he proposed making a $500/month contribution, which combined with other sponsors could have a full-time person working on pythonnet. I took a look through their PR and the changes seem to roughly fit into one of these categories: stylistic (e.g. int vs. var) version change (1.0 vs. 2.4) correctness checks & better support for debugging (e.g. TRACE_ALLOC, FINALIZER_CHECK) code removal due to earlier branch point (domain reload work) additional type support (datetime, timespan) changes to support Python 3? unclear modifications (e.g. removal of initSigs parameter) Are any of these non-contentious? If so, could we cherry-pick those changes at least? Notes Benoit - Felix & Amos are talking on domain reload - next step - write up possible solutions. Working towards consensus, Amos is solving the problem more generally than we are. Likely will be two patches. Victor - If there will be multiple patches, can we split into two PRs? Benoit - that's the goal. Victor - Have two PRs that are ready for review, please review. New PR that introduces performance tests, not yet working with CI. Needs to be split into two PRs: test framework and actual tests. This will enable adding performance tests for any feature. Victor - re Quantconnect PR, can we make some of these changes more generic and bring them to master? Victor will create issues and the plan for PRs to bring them to master. Goal is for Quantconnect to not have to modify base source code. Mark - re Jared's suggestion of $500/month: Unity can't pay someone to maintain Python for .NET without a lot of legal overhead (master service agreement, statement of work, etc). Joining the .NET foundation may help with this. Mark - will move meeting to Thursday to assist Benedikt joining Benedikt- Still didn't hear anything from the .NET Foundation even though I filled their questionnaire now twice Mark - trying to contact .NET foundation from inside Unity, but my contact is on vacation. Will continue when he returns. Benedikt: - I started work on separating the CLR loading out into a separate Python package with implementations for Framework, Core and Mono. This should simplify our build-process a lot. - I started (again) to drop the old csproj files in favour of proper Sdk-style files, will try to finish that after the loader is done - I really like Victor's work on the benchmark, will look into the PR tomorrow. If this turns out well, I'd really like to see where we would get by replacing the P/Invoke style by explicit delegates, as this would finally open up a proper route towards a single Nuget package - After the build-system improvements I'll go forward on the IEnumerable changes started in PR 963. The idea is to drop (almost) all implicit conversions in favour of implementing the correct protocols for .NET objects (leveraging duck typing). - Since I don't use embedding very much, please implement your proposals for domain reload etc. as you please and I'll just have a final look on the respective PRs. I don't have an informed opinion on these cases. Discussion about performance. For good performance you’d need nice C# access to the Buffer protocol: https://docs.python.org/3/c-api/buffer.html <https://docs.python.org/3/c-api/buffer.html> (also available in Python 2.7: https://docs.python.org/2/c-api/buffer.html <https://docs.python.org/2/c-api/buffer.html>) Mohamed- C# method that takes IEnumerable can't be called with Python lists and tuples. Should that work? Worked this way in IronPython, but not ideal. Victor- Have been working on another approach with static list of conversion codecs. You can add your own codec to the list to handle arbitrary conversions. Still WIP, needs design discussion. Benedikt- What about filtering by assembly or class in registration? Victor- could achieve that by subclassing a codec and adding a filter for assembly/class. Discussion on corner cases around automatic conversion of parameters. Implicit conversion is tricky. Mohamed- is there a way to register at runtime? (with your conversion API) Victor- yes, registration is via singleton method. Worth some discussion. Benedikt- would it be better to always require function-based registration? i.e. argument & return type conversion? Discussion on parameter conversion between Python and C#. Non-trivial to do automatic conversion, too many corner cases. Amos- showed proposal for conversion API Victor- how to handle generic types? will send Amos example. Benedikt- if anyone has time to look into Github Actions, they are a potential way to have a single way to run CI on both Windows and *nix. Action Items Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Benoit - document my understanding of soft restart, work with Felix & Amos to get to a shared understanding. Goal is two PRs for domain reload. Victor - will create Github issues for items in the Quantconnect PR and figure out plan for bringing more generic solutions to master. Mark - move meeting to Thursday, same time to accommodate Benedikt's schedule ? - can someone research how hard it would be to move the CI to Github Actions? Victor- send complex type conversion example to Amos Google doc here: https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE… <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…> (please reply or comment in the google doc if I have made any mistakes or missed details) Unless otherwise noted, the next working group meeting will be held on Thursday, November 7, 2019 at 3pm EST. Zoom call link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> Please add agenda items in the google doc above.

1 0

Python for .NET working group meeting notes from Tuesday, September 24, 2019
by Mark Visser 18 Oct '19

18 Oct '19

Attendees: Mark Visser (Unity), Benoit Hudson & Felix Bourbonnais (ImgSpc), Victor Milovanov, Mohamed Koubaa (ANSYS Inc), Amos Li Agenda 1. (Mark) Suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. (Mark) Can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation <https://github.com/pythonnet/pythonnet/issues/956> 3. (Benoit) Python can't reliably re-initialize; new proposal for Unity's domain reload <https://github.com/pythonnet/pythonnet/issues/957> 4. (Benoit) What's the policy with the GIL? Main thread appears to be grabbing the lock and not releasing it. 5. (Mohamed) Can I get clarification on IWrapper vs. reflection for PR #808? Notes Welcome to new contributor Mohamed Koubaa (ANSYS), who is investigating moving his company's platform from IronPython to CPython. 1 & 2) need input from Benedikt, Mark will follow up with Benedikt. 4) Benoit: GIL - Should we expect the GIL to be held after initialization? Victor: I wouldn't expect it to be held. [PyGIL] attribute is informational. 3) Benoit: Python can't reliably re-initialize. last fall's work - force PyFinalize to be called unfortunately doesn't work for native extensions - can't reliably reinitialize proposal instead of calling PyFinalize, delete all links from Python to C# Felix prototyped a solution that tracks all links during proxy generation and cuts all links during domain shutdown to prevent dereference of dead pointers result is just AttributeError instead of crash Amos proposed a different solution which seems to be targeting memory leaks Victor: what about memory leaks? C# side deletes all of its memory If C# holds a reference to Python, it will still exist when C# shuts down - no problem Amos: has a branch that solves one of the problems Benoit & Felix will investigate his branch and see where it intersects with their solution the reason we're using pythonnet is to enable PySide/PyQt and numpy among others 5) Mohamed: Working on a small change to implement the wrapper function for console output via __repr__ https://github.com/pythonnet/pythonnet/pull/808 inheritance over reflection, IWrapper vs. method with a signature Pythonnet can reflect what are the pros and cons? should we go with a convention? Victor: you can take the reflection approach, but need to cache the delegates more of an implementation detail, not actually necessary (note: any errors in the above are mine, feel free to correct any gross misrepresentations I've inadvertently made!) Action Items Mark follow up with Benedikt about Python foundation and using issues to track discussion items Benoit & Felix investigate why GIL is held indefinitely test out Amos's branch and connect over gitter to discuss what parts of the problem each approach is solving Thank you all for attending! The next call will be held at 3pm EST on Tuesday, October 8th, 2019, see you then. Please e-mail me if you're not on the attendee list and I will add you. Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 1

Working group minutes for October 8, 2019
by Mark Visser 08 Oct '19

08 Oct '19

Attendees: Benoit Hudson (ImgSpc), Mohamed Kouba (ANSYS Inc.), Victor Milovanov, Amos Li, Mark Visser (Unity) Agenda (Mark) Suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. Who can tag issues? (Mark) Can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation <https://github.com/pythonnet/pythonnet/issues/956> - Any response? (Mark) PR review (Mark) Issues review Notes Mark - no news on .NET foundation progress, Mark is investigating internally through Unity if there's some way to get the process moving. Benoit - Amos & Felix still need to connect and chat re: PR#957 Benoit - re: grabbing the GIL in issue#964 - this is by design and should stay that way. Need to document implications of multithreaded environment when Python is embedded in a C# application. GIL needs to be explicitly released and regrabbed via BeginAllowThreads/EndAllowThreads otherwise the process will crash. No change is proposed to this behaviour. Victor - will move PRs that aren't ready for merge to draft state so we have a clearer idea of what's actually in process. Mohamed - in IronPython you can have more than one instance running, i.e. more than one scope. Victor - Python 3.7 or 3.8 has/will/may have sub-interpreters working.. Mark/Benoit - RpyC + subprocesses can achieve the same effect but has marshalling/IPC overhead Mohamed - what's the process for supporting a feature like sub-interpreters in pythonnet Victor - basically propose it at a meeting, get alignment on design, write the code Amos - is there any plan for improving performance? Victor - need performance tests before we check in any PRs around performance, no time at the moment. Hopefully in time for the next meeting or shortly after. Action Items Mark - see if there's a path to the .NET foundation internally and get our application unblocked. Benoit - coordinate discussion between Amos & Felix on #957 Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Victor - move PRs that aren't ready for merging to draft status Mohamed - update #808 according to feedback, time permitting Google doc here: https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE… <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…> (please reply or comment in the google doc if I have made any mistakes) Unless otherwise noted, the next working group meeting will be held on Tuesday, October 22 at 3pm EST. Zoom call link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> I have all-day meetings that week so I can't attend. Can someone else fill in as moderator for the call? Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Working group meeting today
by Mark Visser 08 Oct '19

08 Oct '19

Hi everyone, just a reminder that the working group meeting is today, Tuesday October 8th at 3pm EST. Here's the zoom link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Call for discussion items for next week's working group meeting (September 24th)
by Mark Visser 24 Sep '19

24 Sep '19

Hi all, The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST. Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. Mine: 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. Benedikt (filmor(a)gmail.com <mailto:filmor@gmail.com>) can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation <https://github.com/pythonnet/pythonnet/issues/956> cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

3 4

Working group meeting Tuesday September 10th
by Mark Visser 12 Sep '19

12 Sep '19

Hi everyone, Here are the minutes from Tuesday's meeting. Present: Victor Milovanov, Amos Li, Mark Visser Mark mentioned that his team has found a solution to C# domain reload that doesn't crash the Unity Editor. Benoit Hudson will join the next meeting to discuss with the group. Victor has been working on the ability to inherit C# classes from Python classes and override Python methods in C# using attributes. The next meeting will held on Tuesday, September 24th at 3pm EST. If you have not received an invite, please e-mail me directly and I'll add you to the event. Also please note that I went ahead and switched to Zoom for hosting meetings as Google Hangouts is blocked in the PRC. I'll ask the list for agenda items the week before the meeting. Hope to see you all online next time. cheers, -Mark Mark Visser DCC Team Lead Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Memory management when embedding python.
by Thomas Lundgaard Hansen 11 Sep '19

11 Sep '19

Hi. I am embedding Python into a .NET (C#) application. For the purpose of this discussion my code is like the example given here: https://github.com/pythonnet/pythonnet#example (except that I am working with arrays of a substantial size). Each of the "dynamic" variables created in that example are of the .NET type PyObject, which is a disposable type. The code in the example does not call Dispose() on these objects and it would indeed be very inconvenient to do so. Whats the best practice here and what considerations went into this design? Can we expect that the only unmanged resources held by PyObject's are unmanaged memory? (Unless, of course, the PyObject represents a system resource like a file or network stream). Has it been considered to use GC.AddMemoryPressure when PyObjects are backed by a large amount of unmanaged memory? Thanks, Thomas Lundgaard Hansen

3 3

Minutes from meeting Tuesday July 2
by Benedikt Reinartz 02 Jul '19

02 Jul '19

Hi all, Present were: - Amos - Benoit - Benedikt - David - Vasilios - Victor Notes (taken after the meeting, next time I'll be better prepared): - We have decided to (try to) join the .NET foundation, Benedikt will handle the application - Reiterated the current development activities: - Victor will look further into (micro-)benchmarking and encouraged everyone to send in snippets that should not regress in performance - Benedikt outlined again the plan to get platform-independent with a single assembly (may be two, one for Python 2 and one for Python 3) - Victor will follow up again with replacing P/Invoke by explicit delegates after the benchmarking is in place - Unity people are currently busy preparing for SIGRAPH, will be available for more activities, in particular around domain-reload afterwards - Speaking of domain-reloads, Benedikt will look into getting ARM working again by making the domain-reload fixes configurable - Work on integrating QuantConnect's changes will continue, however Benedikt would like to find a way to make the automatic conversions configurable before integrating them as well - Vasilios will contribute test cases for the binding redirect issues he is seeing (https://github.com/pythonnet/pythonnet/issues/832) - The next meeting will take place in two weeks, to actually be able to hear what Amos has to say we'll switch to Skype for the next meeting, the link to the meeting will be sent to the mailing list beforehand If I missed anything, just answer :) Regards Benedikt

1 0