Mailman 3 - Python.NET

Working group notes for January 14, 2021
by Mark Visser 14 Jan '21

14 Jan '21

(I think I forgot to send out the previous meeting notes - you can find them in the Google doc linked at the bottom of this message) Attendees: Benedikt, Mark, Victor, Benoit, Felix, Amos Agenda Review last week's action items: Benedikt will fix 32-bit issues Benedikt to add test that loads unicode function names Felix will submit PR for #1333 (function delegates) Felix will submit PR that rewrites import hook to use pure Python (see #1096 & #727) Notes: 32-bit issues nothing done yet, Benedikt will look into this test that loads unicode function names merged! PR #1329 problem was that we need to make sure we always use our own UTF-8 marshaller we can't use the .NET one because it doesn't exist in all .NET frameworks PR fixes the issue where Benedikt saw it, but there are possibly other places where the same issue exists follow-up issue is #1354 let's link all encoding problems to this issue in Python 3 it's all UTF-8 this particular issue (unicode function names) is fixed function delegates merged! PR #1333 rewrite of import hook to use pure Python (see #1096 & #727) in progress (Felix) rewrite avoids entering C# unless we absolutely need to, because there's no way to prevent C# from "owning" a thread that calls C#, and once C# owns it domain reload hangs waiting for the thread to exit fixes a common pattern "try import PySide; except: import PySide2" - previous implementation hits C# on the first import call, even if PySide doesn't exist internal compiler error (https://github.com/dotnet/roslyn/issues/49760) still present in latest .NET 5.0.2 might still be a Roslyn point release? doesn't help because Roslyn likely frozen until .NET 6 Victor will find out when they plan on releasing a fix required for replacing PInvoke delegates for reference types with using unmanaged function pointers (supposed to work in new C#, but triggers an internal compiler error) blocking our ability to do a multi-stage startup and dynamically bind to the right Python dll/dso/dylib anywhere but Mono (which supports __Internal as the DllImport string) helpful for embedding .NET in Python since want to use the statically linked symbols in the python executable - using a dynamic library, even if present, leads to weird problems we'll find out if a fix is forthcoming and if not, use a workaround Action items: Benedikt will fix 32-bit issues Felix will complete rewrite of import hook to avoid calling C# unless necessary Victor will find out if there will be a Roslyn release fixing the internal compiler error The meeting notes google doc is here <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…>. Feel free to correct or add additional information. The next meeting will be held on Thursday, January 28 at 12pm EST (click for your time zone) <https://www.google.com/search?q=12:00+pm+EST>. Mark Visser Senior Dev Manager, M&E Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

RANGE object access from Python
by AlexM 02 Dec '20

02 Dec '20

Hello, guys, I have a C# object, I need access to it's properties in Python 2.7. I got this type returned from a C# dll: <class 'Philips.CT.Host.ModelAbstraction.DataStructures.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]'> This should be a "Range" data type in C#. Thanks a lot for any suggestion! Alex.

2 2

How to Run Pythinon.net Demo Samples
by Vishal Patel 26 Nov '20

26 Nov '20

Hi, I need your help regarding how to run Pythinon.net Demo samples (C:\Program Files\PythonNet\demo) in c#. Best Regards, Vishal Patel | PROJECT LEADER Openxcell Email Id: seo4(a)openxcelltechnolabs.com Website: http://www.openxcell.com

1 0

Allowing Python.Net to run with latest Python version
by Vince Luff 26 Nov '20

26 Nov '20

Hello, I work on a commercial Python package which I am currently enhancing to use Python.Net in order to access some .Net libraries. Before using Python.Net we specified to end users that the requirement for running our package was "Python 3 installed". Now that we are using Python.Net we can't say that statement because Python.Net 2.5.1 supports Python 3.8 but not Python 3.9. Since our package is not a stand-alone application, but it instead provides a Python API for users Python scripts to call, it's not an option for us to convert the whole delivery into an exe using something like PyInstaller. So I'm looking for advice about the best option: 1. Specify that end users install a specific version of Python (3.8) in order to use our application. 2. Include Python 3.8 in our installer and install this if it's not already installed. Is this a good idea? 3. Any other ideas? Many thanks, DareDevilDenis

3 2

Working group notes for November 5, 2020
by Mark Visser 19 Nov '20

19 Nov '20

Attendees: Victor, Benedikt, Mark, Amos, Benoit, Felix Agenda Review last week's action items: Felix & Benoit to add tests to master in preparation for the soft reload fix PR - will be ignored for now (or expected to fail - ideal if that's possible) Victor to review these to understand the need for isolated testing environment for each test discuss soft init/reload/shutdown modes: what are the use cases for each? are they all necessary? need to document if so. (Benoit/Felix) Notes: PR for soft reload tests is failing, hasn't run on travis coverage reported is not accurate let's not worry too much about the coverage numbers and try to provide good tests for new features build times have gotten slower over time due to testing all shutdown modes do we need to run full tests for all modes? some of this is likely redundant let's remove unnecessary tests from travis additional run for every mode github actions? we should keep at least one version of python testing all shutdown modes drop everything except 3.7 from testing shutdown since that's the version we support in Unity? still want to catch regressions caused by new Python versions (e.g. API changes) long term we should look at moving from travis to github actions Unity shutdown/domain reload is using reload mode. do we need other two modes? (soft / none) we never use reloading when embedding C# in Python - just kill workers/instances and restart can we remove soft & none modes? what are the use cases for the other two modes? reload mode requires implementing a customer serializer for CLR objects Feilx is working on making the custom serializer optional so it still works without it with this update reload won't fail without a serializer does this make soft mode redundant? there are a lot of places where there are three code paths depending on the shutdown mode - can we simplify soft mode has no such requirement none - used for shutting down Python and not restarting it - makes no attempt to save state soft - does not maintain connections between CLR and Python objects, but saves state of Python interpreter reload - also maintains connections between CLR and Python objects all three modes are useful as they facilitate testing different parts of the system e.g. if something fails on only one of the modes, it's useful data currently all tests are run with all modes in all versions Travis problems jobs take forever to start job status not reporting at least partly connected to new free tier limitations https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing <https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing>  we will need to move things off travis-ci.org by the end of the year travis is shutting down free travis access due to abuse move to travis-ci.com, still potential for free use but will be potentially limited longer term move to github actions needs to be evaluated Benedikt's SDK-style build system branch working well, except for some unexpected crashes using poetry instead of setuptools - much simpler than current PRs to come Action items: Felix will reduce the test combinations for soft shutdown on appveyor and travis to reduce build times and make a PR Felix will look at extracting domain reload tests to reduce/simplify build times and report back Benedikt will fix 32-bit issues The meeting notes google doc is here <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…>. Feel free to correct or add additional information. The next meeting will be held on Thursday, December 3 at 12pm EST (click for your time zone) <https://www.google.com/search?q=12:00+pm+EST>. Mark Visser Senior Dev Manager, M&E Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

3 2

Working group notes for October 22, 2020
by Victor “LOST” Milovanov 22 Oct '20

22 Oct '20

1 0

Arrays float
by Gavin 25 Sep '20

25 Sep '20

Hello im trying to convert this line of code buffer = array.array("f", [0.0]) * (WIDTH * HEIGHT * bufferdepth) but have problems its an image buffer of floats but it does not like the [0,0] can anyone helpthanks in advance

2 1

Working group notes for September 10, 2020
by Victor “LOST” Milovanov 10 Sep '20

10 Sep '20

1 0

Working group notes for August 13, 2020
by Mark Visser 13 Aug '20

13 Aug '20

Attendees: Mark, Benoit, Felix, Mohamed, Victor, Amos Agenda Review action items from last meeting: Benedikt to confirm WITH_DEBUG builds need to be updated for 3.9 (low priority) and will create issue to document for later Benedikt - Add reference to backports-2.5 branch in README Benedikt - Switch to .NET Standard for everything but ClrModule Benedikt - will look at Amos's fix for #891 issue in PR #958 Benedikt - will look into Python Buffer branch again Benedikt to look into weirdness with pycparser in appveyor.yml (see PR #1175 <https://github.com/pythonnet/pythonnet/pull/1175>) Domain reload update Notes Felix has been testing domain reload branch mostly success with a few fixes needed issues if you register a Python function as a delegate directly into a C# event, it is not retained - currently throws an exception due to leaving the Python function in a bad state would be ideal to bring back the connection Acceptable to lose the connection, but must leave python.net in a good state otherwise if you maintain a proxy to a C# object in Python, on domain reload the proxy usually gives several different exceptions and sometimes crashes correct behaviour is to raise an exception, ideally the same one with no crashes if an assembly/type/class member/etc is missing after domain reload, causes problems on the Python side needs to be gracefully handled finalizers at exit fix done by Felix PR not yet made other than these issues reload mode works really well soft shutdown mode doesn't work well for domain reload it unloads all modules, which doesn't work for native modules (by design) intended for use in the game, not the editor Victor approved landing it to master once remaining issues raised in PR are fixed Felix/Benoit to fix the minor issues in the PR and land the patch If it can't land to master, Victor will create a branch for it in the repo Remaining issues found by Felix can land to master after the main PR lands to master. Process: Fixes: Felix submits a PR for the fix (ideally with test case) Issues: create an issue, Felix submits PR with test case, invites Amos to fix. Amos fixes it. Land PR with test case + fix to master. Documentation would be good Proposal: Unity/imaginary spaces writes an outline, amos fills in outline, Unity/imaginary spaces revises it Python 2 support for soft shutdown / reload is not necessary. Action Items Benedikt to confirm WITH_DEBUG builds need to be updated for 3.9 (low priority) and will create issue to document for later Benedikt - Add reference to backports-2.5 branch in README Benedikt - Switch to .NET Standard for everything but ClrModule Benedikt - will look at Amos's fix for #891 issue in PR #958 Benedikt - will look into Python Buffer branch again Benedikt to look into weirdness with pycparser in appveyor.yml (see PR #1175 <https://github.com/pythonnet/pythonnet/pull/1175>) Felix to create test case for Amos that demonstrates the "Python function as delegate fails on reload" (and others) and log issue - will create unit tests to land after main PR lands Benoit/Felix to create doc outline for reload & soft shutdown modes (developer docs) Felix will fix remaining domain reload issues raised by Victor Amos will fix new issues to land after main PR lands The meeting notes google doc is here <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…>. Feel free to correct or add additional information. The next meeting will be held on Thursday, August 27 at 12pm EST (click for your time zone) <https://www.google.com/search?q=12:00+pm+EST>. Mark Visser M&E Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0

Working group notes for July 30, 2020
by Mark Visser 30 Jul '20

30 Jul '20

Attendees: Amos, Victor, Mark, Benedikt, Mohamed Agenda Review action items from last meeting Benedikt - Check whether we need to change the `WITH_DEBUG` variant of PyObject_TYPE for newer versions of Python (> 3.8?) Benedikt - Add reference to backports-2.5 branch in README Benedikt - Switch to .NET Standard for everything but ClrModule Benedikt - Change project files to new format Victor - Consider adding a StolenReference type Benedikt - will look at Amos's fix for #891 issue in PR #958 Benedikt - will look into Python Buffer branch again Soft shutdown PR Notes Check whether we need to change the `WITH_DEBUG` variant of PyObject_TYPE for newer versions of Python (> 3.8?) in newer versions of Python WITH_DEBUG builds have the same layout we need to remove the special case not high priority right now since it's 3.9+ Switch to .NET Standard for everything but ClrModule Benedikt will complete today to unblock Mohamed Change project files to new format need to get rid of old project files and replace by new ones CLR module needs to stay on old format StolenReference type created issue for StolenReference type <https://github.com/pythonnet/pythonnet/issues/1193> Python Buffer branch again up for grabs if someone else wants to contribute is working fine - was held back until release because it's a Python 3-only branch should be ready to be merged once rebased Soft shutdown PR Unity will test before fall Python 3.9 multiphase initialization for modules used for subinterpreter support among other things every module is an object instead of a singleton can have multiple instances of a module thus there can be multiple instances of the clr module can be opted out of - see PEP 489 <https://www.python.org/dev/peps/pep-0489/> (other PEPs are based on it as well) may incur work on Python.NET side to support or disable PEP 554 <https://www.python.org/dev/peps/pep-0554/> also relevant (subinterpreters python API for 3.10) interesting for embedding use case of embedding Python into .NET nicer API seems like a good idea to leverage this for our own embedding API needs investigation Discussion on broken appveyor.yml fixed in PR #1175 <https://github.com/pythonnet/pythonnet/pull/1175> Action Items Benedikt to confirm WITH_DEBUG builds need to be updated for 3.9 (low priority) and will create issue to document for later Benedikt - Add reference to backports-2.5 branch in README Benedikt - Switch to .NET Standard for everything but ClrModule Benedikt - will look at Amos's fix for #891 issue in PR #958 Benedikt - will look into Python Buffer branch again Benedikt to look into weirdness with pycparser in appveyor.yml (see PR #1175 <https://github.com/pythonnet/pythonnet/pull/1175>) The meeting notes google doc is here <https://docs.google.com/document/d/1rJVU84B_dgx58-_EopjRtOJVFAI2WfHJYV0n7uE…>. Feel free to correct or add additional information. The next meeting will be held on Thursday, August 13 at 12pm EST (click for your time zone) <https://www.google.com/search?q=12:00+pm+EST>. Mark Visser M&E Dev Manager Unity Technologies - www.unity3d.com <http://www.unity3d.com/>

1 0