Mailman 3 October 2019 - Python.NET

Working group minutes for October 22, 2019
by Mark Visser 22 Oct '19

22 Oct '19

Attendees: Benedikt Reinhartz (Project Lead), Benoit Hudson (ImgSpc), Felix Bourbonnais (ImgSpc), Mohamed Kouba (ANSYS Inc.), Victor Milovanov, Amos Li, Mark Visser (Unity) Agenda (Mark) I got a reply from Jared Broad (QuantConnect): His team is unable to devote resources to anything other than the PR they agreed to open (#895). He wanted to remind us that he proposed making a $500/month contribution, which combined with other sponsors could have a full-time person working on pythonnet. I took a look through their PR and the changes seem to roughly fit into one of these categories: stylistic (e.g. int vs. var) version change (1.0 vs. 2.4) correctness checks & better support for debugging (e.g. TRACE_ALLOC, FINALIZER_CHECK) code removal due to earlier branch point (domain reload work) additional type support (datetime, timespan) changes to support Python 3? unclear modifications (e.g. removal of initSigs parameter) Are any of these non-contentious? If so, could we cherry-pick those changes at least? Notes Benoit - Felix & Amos are talking on domain reload - next step - write up possible solutions. Working towards consensus, Amos is solving the problem more generally than we are. Likely will be two patches. Victor - If there will be multiple patches, can we split into two PRs? Benoit - that's the goal. Victor - Have two PRs that are ready for review, please review. New PR that introduces performance tests, not yet working with CI. Needs to be split into two PRs: test framework and actual tests. This will enable adding performance tests for any feature. Victor - re Quantconnect PR, can we make some of these changes more generic and bring them to master? Victor will create issues and the plan for PRs to bring them to master. Goal is for Quantconnect to not have to modify base source code. Mark - re Jared's suggestion of $500/month: Unity can't pay someone to maintain Python for .NET without a lot of legal overhead (master service agreement, statement of work, etc). Joining the .NET foundation may help with this. Mark - will move meeting to Thursday to assist Benedikt joining Benedikt- Still didn't hear anything from the .NET Foundation even though I filled their questionnaire now twice Mark - trying to contact .NET foundation from inside Unity, but my contact is on vacation. Will continue when he returns. Benedikt: - I started work on separating the CLR loading out into a separate Python package with implementations for Framework, Core and Mono. This should simplify our build-process a lot. - I started (again) to drop the old csproj files in favour of proper Sdk-style files, will try to finish that after the loader is done - I really like Victor's work on the benchmark, will look into the PR tomorrow. If this turns out well, I'd really like to see where we would get by replacing the P/Invoke style by explicit delegates, as this would finally open up a proper route towards a single Nuget package - After the build-system improvements I'll go forward on the IEnumerable changes started in PR 963. The idea is to drop (almost) all implicit conversions in favour of implementing the correct protocols for .NET objects (leveraging duck typing). - Since I don't use embedding very much, please implement your proposals for domain reload etc. as you please and I'll just have a final look on the respective PRs. I don't have an informed opinion on these cases. Discussion about performance. For good performance you’d need nice C# access to the Buffer protocol: https://docs.python.org/3/c-api/buffer.html <https://docs.python.org/3/c-api/buffer.html> (also available in Python 2.7: https://docs.python.org/2/c-api/buffer.html <https://docs.python.org/2/c-api/buffer.html>) Mohamed- C# method that takes IEnumerable can't be called with Python lists and tuples. Should that work? Worked this way in IronPython, but not ideal. Victor- Have been working on another approach with static list of conversion codecs. You can add your own codec to the list to handle arbitrary conversions. Still WIP, needs design discussion. Benedikt- What about filtering by assembly or class in registration? Victor- could achieve that by subclassing a codec and adding a filter for assembly/class. Discussion on corner cases around automatic conversion of parameters. Implicit conversion is tricky. Mohamed- is there a way to register at runtime? (with your conversion API) Victor- yes, registration is via singleton method. Worth some discussion. Benedikt- would it be better to always require function-based registration? i.e. argument & return type conversion? Discussion on parameter conversion between Python and C#. Non-trivial to do automatic conversion, too many corner cases. Amos- showed proposal for conversion API Victor- how to handle generic types? will send Amos example. Benedikt- if anyone has time to look into Github Actions, they are a potential way to have a single way to run CI on both Windows and *nix. Action Items Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Benoit - document my understanding of soft restart, work with Felix & Amos to get to a shared understanding. Goal is two PRs for domain reload. Victor - will create Github issues for items in the Quantconnect PR and figure out plan for bringing more generic solutions to master. Mark - move meeting to Thursday, same time to accommodate Benedikt's schedule ? - can someone research how hard it would be to move the CI to Github Actions? Victor- send complex type conversion example to Amos Google doc here: https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE… <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…> (please reply or comment in the google doc if I have made any mistakes or missed details) Unless otherwise noted, the next working group meeting will be held on Thursday, November 7, 2019 at 3pm EST. Zoom call link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> Please add agenda items in the google doc above.

1 0

Python for .NET working group meeting notes from Tuesday, September 24, 2019
by Mark Visser 18 Oct '19

18 Oct '19

Attendees: Mark Visser (Unity), Benoit Hudson & Felix Bourbonnais (ImgSpc), Victor Milovanov, Mohamed Koubaa (ANSYS Inc), Amos Li Agenda 1. (Mark) Suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. (Mark) Can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation <https://github.com/pythonnet/pythonnet/issues/956> 3. (Benoit) Python can't reliably re-initialize; new proposal for Unity's domain reload <https://github.com/pythonnet/pythonnet/issues/957> 4. (Benoit) What's the policy with the GIL? Main thread appears to be grabbing the lock and not releasing it. 5. (Mohamed) Can I get clarification on IWrapper vs. reflection for PR #808? Notes Welcome to new contributor Mohamed Koubaa (ANSYS), who is investigating moving his company's platform from IronPython to CPython. 1 & 2) need input from Benedikt, Mark will follow up with Benedikt. 4) Benoit: GIL - Should we expect the GIL to be held after initialization? Victor: I wouldn't expect it to be held. [PyGIL] attribute is informational. 3) Benoit: Python can't reliably re-initialize. last fall's work - force PyFinalize to be called unfortunately doesn't work for native extensions - can't reliably reinitialize proposal instead of calling PyFinalize, delete all links from Python to C# Felix prototyped a solution that tracks all links during proxy generation and cuts all links during domain shutdown to prevent dereference of dead pointers result is just AttributeError instead of crash Amos proposed a different solution which seems to be targeting memory leaks Victor: what about memory leaks? C# side deletes all of its memory If C# holds a reference to Python, it will still exist when C# shuts down - no problem Amos: has a branch that solves one of the problems Benoit & Felix will investigate his branch and see where it intersects with their solution the reason we're using pythonnet is to enable PySide/PyQt and numpy among others 5) Mohamed: Working on a small change to implement the wrapper function for console output via __repr__ https://github.com/pythonnet/pythonnet/pull/808 inheritance over reflection, IWrapper vs. method with a signature Pythonnet can reflect what are the pros and cons? should we go with a convention? Victor: you can take the reflection approach, but need to cache the delegates more of an implementation detail, not actually necessary (note: any errors in the above are mine, feel free to correct any gross misrepresentations I've inadvertently made!) Action Items Mark follow up with Benedikt about Python foundation and using issues to track discussion items Benoit & Felix investigate why GIL is held indefinitely test out Amos's branch and connect over gitter to discuss what parts of the problem each approach is solving Thank you all for attending! The next call will be held at 3pm EST on Tuesday, October 8th, 2019, see you then. Please e-mail me if you're not on the attendee list and I will add you. Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 1

Working group minutes for October 8, 2019
by Mark Visser 08 Oct '19

08 Oct '19

Attendees: Benoit Hudson (ImgSpc), Mohamed Kouba (ANSYS Inc.), Victor Milovanov, Amos Li, Mark Visser (Unity) Agenda (Mark) Suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. Who can tag issues? (Mark) Can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation <https://github.com/pythonnet/pythonnet/issues/956> - Any response? (Mark) PR review (Mark) Issues review Notes Mark - no news on .NET foundation progress, Mark is investigating internally through Unity if there's some way to get the process moving. Benoit - Amos & Felix still need to connect and chat re: PR#957 Benoit - re: grabbing the GIL in issue#964 - this is by design and should stay that way. Need to document implications of multithreaded environment when Python is embedded in a C# application. GIL needs to be explicitly released and regrabbed via BeginAllowThreads/EndAllowThreads otherwise the process will crash. No change is proposed to this behaviour. Victor - will move PRs that aren't ready for merge to draft state so we have a clearer idea of what's actually in process. Mohamed - in IronPython you can have more than one instance running, i.e. more than one scope. Victor - Python 3.7 or 3.8 has/will/may have sub-interpreters working.. Mark/Benoit - RpyC + subprocesses can achieve the same effect but has marshalling/IPC overhead Mohamed - what's the process for supporting a feature like sub-interpreters in pythonnet Victor - basically propose it at a meeting, get alignment on design, write the code Amos - is there any plan for improving performance? Victor - need performance tests before we check in any PRs around performance, no time at the moment. Hopefully in time for the next meeting or shortly after. Action Items Mark - see if there's a path to the .NET foundation internally and get our application unblocked. Benoit - coordinate discussion between Amos & Felix on #957 Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter Victor - move PRs that aren't ready for merging to draft status Mohamed - update #808 according to feedback, time permitting Google doc here: https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE… <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…> (please reply or comment in the google doc if I have made any mistakes) Unless otherwise noted, the next working group meeting will be held on Tuesday, October 22 at 3pm EST. Zoom call link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> I have all-day meetings that week so I can't attend. Can someone else fill in as moderator for the call? Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Working group meeting today
by Mark Visser 08 Oct '19

08 Oct '19

Hi everyone, just a reminder that the working group meeting is today, Tuesday October 8th at 3pm EST. Here's the zoom link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0