[Python.NET] Working group minutes for October 22, 2019

Tue Oct 22 16:21:48 EDT 2019

Attendees:  Benedikt Reinhartz (Project Lead), Benoit Hudson (ImgSpc), Felix Bourbonnais (ImgSpc), Mohamed Kouba (ANSYS Inc.), Victor Milovanov, Amos Li, Mark Visser (Unity)

Agenda
(Mark) I got a reply from Jared Broad (QuantConnect): His team is unable to devote resources to anything other than the PR they agreed to open (#895). He wanted to remind us that he proposed making a $500/month contribution, which combined with other sponsors could have a full-time person working on pythonnet.

I took a look through their PR and the changes seem to roughly fit into one of these categories:
stylistic (e.g. int vs. var)
version change (1.0 vs. 2.4)
correctness checks & better support for debugging (e.g. TRACE_ALLOC, FINALIZER_CHECK)
code removal due to earlier branch point (domain reload work)
additional type support (datetime, timespan)
changes to support Python 3?
unclear modifications (e.g. removal of initSigs parameter)

Are any of these non-contentious? If so, could we cherry-pick those changes at least?

Notes

Benoit - Felix & Amos are talking on domain reload - next step - write up possible solutions. Working towards consensus, Amos is solving the problem more generally than we are. Likely will be two patches. 
Victor - If there will be multiple patches, can we split into two PRs?
Benoit - that's the goal.
Victor - Have two PRs that are ready for review, please review. New PR that introduces performance tests, not yet working with CI. Needs to be split into two PRs: test framework and actual tests. This will enable adding performance tests for any feature.

Victor - re Quantconnect PR, can we make some of these changes more generic and bring them to master? Victor will create issues and the plan for PRs to bring them to master. Goal is for Quantconnect to not have to modify base source code.

Mark - re Jared's suggestion of $500/month: Unity can't pay someone to maintain Python for .NET without a lot of legal overhead (master service agreement, statement of work, etc). Joining the .NET foundation may help with this.

Mark - will move meeting to Thursday to assist Benedikt joining

Benedikt- Still didn't hear anything from the .NET Foundation even though I filled their questionnaire now twice
Mark - trying to contact .NET foundation from inside Unity, but my contact is on vacation. Will continue when he returns.

Benedikt:
- I started work on separating the CLR loading out into a separate Python package with implementations for Framework, Core and Mono. This should simplify our build-process a lot.
- I started (again) to drop the old csproj files in favour of proper Sdk-style files, will try to finish that after the loader is done
- I really like Victor's work on the benchmark, will look into the PR tomorrow. If this turns out well, I'd really like to see where we would get by replacing the P/Invoke style by explicit delegates, as this would finally open up a proper route towards a single Nuget package
- After the build-system improvements I'll go forward on the IEnumerable changes started in PR 963. The idea is to drop (almost) all implicit conversions in favour of implementing the correct protocols for .NET objects (leveraging duck typing).
- Since I don't use embedding very much, please implement your proposals for domain reload etc. as you please and I'll just have a final look on the respective PRs. I don't have an informed opinion on these cases.

Discussion about performance. For good performance you’d need nice C# access to the Buffer protocol: https://docs.python.org/3/c-api/buffer.html <https://docs.python.org/3/c-api/buffer.html> (also available in Python 2.7: https://docs.python.org/2/c-api/buffer.html <https://docs.python.org/2/c-api/buffer.html>)

Mohamed- C# method that takes IEnumerable can't be called with Python lists and tuples. Should that work? Worked this way in IronPython, but not ideal.
Victor- Have been working on another approach with static list of conversion codecs. You can add your own codec to the list to handle arbitrary conversions. Still WIP, needs design discussion.
Benedikt- What about filtering by assembly or class in registration?
Victor- could achieve that by subclassing a codec and adding a filter for assembly/class.
Discussion on corner cases around automatic conversion of parameters. Implicit conversion is tricky.
Mohamed- is there a way to register at runtime? (with your conversion API)
Victor- yes, registration is via singleton method. Worth some discussion.
Benedikt- would it be better to always require function-based registration? i.e. argument & return type conversion?
Discussion on parameter conversion between Python and C#. Non-trivial to do automatic conversion, too many corner cases.
Amos- showed proposal for conversion API
Victor- how to handle generic types? will send Amos example.

Benedikt- if anyone has time to look into Github Actions, they are a potential way to have a single way to run CI on both Windows and *nix.

Action Items
Benoit - document how to use pythonnet when you expect to run multithreaded python code (BeginAllowThreads/EndAllowThreads) in an embedded interpreter 
Benoit - document my understanding of soft restart, work with Felix & Amos to get to a shared understanding. Goal is two PRs for domain reload.
Victor - will create Github issues for items in the Quantconnect PR and figure out plan for bringing more generic solutions to master.
Mark - move meeting to Thursday, same time to accommodate Benedikt's schedule
? - can someone research how hard it would be to move the CI to Github Actions?
Victor- send complex type conversion example to Amos

Google doc here: https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE1Oak/edit# <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE1Oak/edit#>

(please reply or comment in the google doc if I have made any mistakes or missed details)

Unless otherwise noted, the next working group meeting will be held on Thursday, November 7, 2019 at 3pm EST. Zoom call link: https://unity3d.zoom.us/j/346931511 <https://unity3d.zoom.us/j/346931511> 
Please add agenda items in the google doc above.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20191022/96278f60/attachment-0001.html>