From info at egenix.com Mon Jul 1 08:59:13 2024 From: info at egenix.com (eGenix Team) Date: Mon, 1 Jul 2024 14:59:13 +0200 Subject: ANN: eGenix PyRun - One file Python Runtime 2.5.0 Message-ID: <7afff6ed-505b-4915-bfae-059be2b3fabf@egenix.com> *ANNOUNCING* eGenix PyRun - One file Python Runtime Version 2.5.0 Python runtime taking up just 4-6MB on disk This announcement is also available on our web-site for online reading: https://www.egenix.com/company/news/eGenix-PyRun-2.5.0-GA.html ------------------------------------------------------------------------ *INTRODUCTION* *eGenix PyRun*? is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run Python based scripts and applications to Unix based systems simple and efficient. eGenix PyRun's executable only needs 4-6MB on disk, but still supports most Python applications and scripts. Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to containers, VMs, clusters, client installations, customers or end-users. It makes "installing" Python on a Unix based system as simple as copying a single file. eGenix has been using eGenix PyRun as run-time for the Linux version of mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product. We provide the source archive to build your own *eGenix PyRun on Github* , as well as a few binary distributions to get you started on Linux x86_64. In the future, we will set up automated builds for several other platforms. Please see the product page for more details: >>> eGenix PyRun - One file Python Runtime ------------------------------------------------------------------------ *NEWS* This major release of eGenix PyRun comes with the following enhancements: * Added support for *Python 3.8 - 3.11* * Removed support for Python 3.5-3.7 * Modernized the directory setup and build * Changed the license to the *Apache2 license* * Extracted the code from our internal mono-repo to put on Github * *Relaunched the project on Github* For a complete list of changes, please see the *eGenix PyRun Changelog *. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jul 01 2024) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From me at sc1f1dan.com Thu Jul 4 03:51:34 2024 From: me at sc1f1dan.com (Daniel) Date: Thu, 04 Jul 2024 00:51:34 -0700 Subject: python for irc client Message-ID: <87msmx4mt5.fsf@raspberrypi> Hi guys - I have historical experience developing sofwtare for my own use. It has been quite a while since doing so and the advent of new languages has brought me here. Python has built quite a reputation. It would be fun to pick up a new language while I'm at it. I've been a consumer of IRC since the nineties and have been running an instance of quassel core on an old laptop for the last decade. Over the years, my use of xwindows has dramatically decreased and I spend 90% of my computer time with multiple panes of tmux while I do my usual daily fun. One thing missing is a good textmode irc client that will connect to quassel core. I've seen efforts to make a plugin for weechat but, to date, I don't see much progress on that end. In your wisdom, would python be a good environment to accomplish this? I'd likely use extended ascii and colors. The point would be to minimize the memory footprint of the application. I don't use standard desktop computers anymore - I'm writing this on my beloved pi400 using emacs. Thanks Daniel From inhahe at gmail.com Thu Jul 4 05:22:53 2024 From: inhahe at gmail.com (inhahe) Date: Thu, 4 Jul 2024 05:22:53 -0400 Subject: python for irc client In-Reply-To: <87msmx4mt5.fsf@raspberrypi> References: <87msmx4mt5.fsf@raspberrypi> Message-ID: On Thu, Jul 4, 2024 at 5:14?AM Daniel via Python-list < python-list at python.org> wrote: > Hi guys - > > I have historical experience developing sofwtare for my own use. It has > been > quite a while since doing so and the advent of new languages has brought me > here. Python has built quite a reputation. It would be fun to pick up a > new language while I'm at it. > > I've been a consumer of IRC since the nineties and have been running an > instance of quassel core on an old laptop for the last decade. Over the > years, my use of xwindows has dramatically decreased and I spend 90% of my > computer time with multiple panes of tmux while I do my usual daily fun. > One > thing missing is a good textmode irc client that will connect to quassel > core. > > I've seen efforts to make a plugin for weechat but, to date, I don't see > much > progress on that end. > > In your wisdom, would python be a good environment to accomplish this? I'd > likely use extended ascii and colors. The point would be to minimize the > memory footprint of the application. > > I don't use standard desktop computers anymore - I'm writing this on my > beloved pi400 using emacs. > > Thanks > > Daniel > -- > https://mail.python.org/mailman/listinfo/python-list I think Python would be a great language to write an IRC client in, it's a rapid-development language, and also Python is particularly good for text manipulation and the IRC protocol is textual rather than binary. But, if your only purpose for using Python is to reduce the memory footprint, I'm not sure. I don't know specifically, but I'd guess Python has a higher memory footprint than, say, C, because it's a high-level language. For example, each variable has to be boxed, and also the interpreter has to be loaded.. Regarding high ASCII, I don't know if that works in IRC, but either way, ASCII isn't really enough nowadays. You need to support Unicode; specifically, UTF-8. From inhahe at gmail.com Thu Jul 4 05:25:48 2024 From: inhahe at gmail.com (inhahe) Date: Thu, 4 Jul 2024 05:25:48 -0400 Subject: python for irc client In-Reply-To: References: <87msmx4mt5.fsf@raspberrypi> Message-ID: On Thu, Jul 4, 2024 at 5:22?AM inhahe wrote: > > > On Thu, Jul 4, 2024 at 5:14?AM Daniel via Python-list < > python-list at python.org> wrote: > >> >> In your wisdom, would python be a good environment to accomplish this? > > > I think Python would be a great language to write an IRC client in, it's a > rapid-development language, and also Python is particularly good for text > manipulation and the IRC protocol is textual rather than binary. > Oh yeah, I forgot I was going to mention that Twisted has already done a lot of the dirty work for you if you make it in Python...they have twisted. words.protocols.irc, which implements the IRC protocol. (I don't know if it's up to date and supports ircv3, though.) From olegsivokon at gmail.com Thu Jul 4 09:55:09 2024 From: olegsivokon at gmail.com (Left Right) Date: Thu, 4 Jul 2024 15:55:09 +0200 Subject: python for irc client In-Reply-To: References: <87msmx4mt5.fsf@raspberrypi> Message-ID: Hi. Just FYI, I use Erc (in Emacs). I'm not a very advanced user, perhaps, but I never felt like I miss anything. That's not to stop you from making your own, but if you just need a decent text client for IRC, then there's already at least one. On Thu, Jul 4, 2024 at 11:30?AM inhahe via Python-list wrote: > > On Thu, Jul 4, 2024 at 5:22?AM inhahe wrote: > > > > > > > On Thu, Jul 4, 2024 at 5:14?AM Daniel via Python-list < > > python-list at python.org> wrote: > > > >> > >> In your wisdom, would python be a good environment to accomplish this? > > > > > > > I think Python would be a great language to write an IRC client in, it's a > > rapid-development language, and also Python is particularly good for text > > manipulation and the IRC protocol is textual rather than binary. > > > > Oh yeah, I forgot I was going to mention that Twisted has already done a > lot of the dirty work for you if you make it in Python...they have twisted. > words.protocols.irc, which implements the IRC protocol. (I don't know if > it's up to date and supports ircv3, though.) > -- > https://mail.python.org/mailman/listinfo/python-list From me at sc1f1dan.com Thu Jul 4 16:43:33 2024 From: me at sc1f1dan.com (Daniel) Date: Thu, 04 Jul 2024 13:43:33 -0700 Subject: python for irc client References: <87msmx4mt5.fsf@raspberrypi> Message-ID: <87jzi0vqfe.fsf@raspberrypi> inhahe writes: > On Thu, Jul 4, 2024 at 5:14?AM Daniel via Python-list < > python-list at python.org> wrote: > >> Hi guys - >> >> I have historical experience developing sofwtare for my own use. It has >> been >> quite a while since doing so and the advent of new languages has brought me >> here. Python has built quite a reputation. It would be fun to pick up a >> new language while I'm at it. >> >> I've been a consumer of IRC since the nineties and have been running an >> instance of quassel core on an old laptop for the last decade. Over the >> years, my use of xwindows has dramatically decreased and I spend 90% of my >> computer time with multiple panes of tmux while I do my usual daily fun. >> One >> thing missing is a good textmode irc client that will connect to quassel >> core. >> >> I've seen efforts to make a plugin for weechat but, to date, I don't see >> much >> progress on that end. >> >> In your wisdom, would python be a good environment to accomplish this? I'd >> likely use extended ascii and colors. The point would be to minimize the >> memory footprint of the application. >> >> I don't use standard desktop computers anymore - I'm writing this on my >> beloved pi400 using emacs. >> >> Thanks >> >> Daniel >> -- >> https://mail.python.org/mailman/listinfo/python-list > > > I think Python would be a great language to write an IRC client in, it's a > rapid-development language, and also Python is particularly good for text > manipulation and the IRC protocol is textual rather than binary. But, if > your only purpose for using Python is to reduce the memory footprint, I'm > not sure. I don't know specifically, but I'd guess Python has a higher > memory footprint than, say, C, because it's a high-level language. For > example, each variable has to be boxed, and also the interpreter has to be > loaded.. > > Regarding high ASCII, I don't know if that works in IRC, but either way, > ASCII isn't really enough nowadays. You need to support Unicode; > specifically, UTF-8. Okay great. Since my original post, I settled on UTF8. I have to create a list of requirements for v1.0 to limit scope creep and I can actually get this done. I may put it on github and solicit for assistance at some point. Thanks for the response, both of them. I'll look at the other code and see how I can fold it in. What I have to find out, still, is how the core server manages the messages. I suspect the core does all the sending and receiving and the client just sends the packets to core for management. That's just a guess though. I still have to review the liraries, this is a new idea hatched last night so I have yet to investigate much. My initial thought was C++ but this would be my first termianl-only application in many years so I thought a different coding platform would be effective. Daniel From rob.cliffe at btinternet.com Sat Jul 6 06:49:06 2024 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 6 Jul 2024 11:49:06 +0100 Subject: Best use of "open" context manager Message-ID: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Consider this scenario (which I ran into in real life): ??? I want to open a text file and do a lot of processing on the lines of that file. ??? If the file does not exist I want to take appropriate action, e.g. print an error message and abort the program. I might write it like this: try: ??? with open(FileName) as f: ??? ??? for ln in f: ??? ??? ??? print("I do a lot of processing here") ??? ??? ??? # Many lines of code here ..... except FileNotFoundError: ??? print(f"File {FileName} not found") ??? sys.exit() but this violates the principle that a "try" suite should be kept small, so that only targeted exceptions are trapped, not to mention that having "try" and "except" far apart decreases readability. Or I might write it like this: try: ??? f = open(FileName) as f: ??? FileLines = f.readlines() except FileNotFoundError: ??? print(f"File {FileName} not found") ??? sys.exit() # I forgot to put "f.close()" here -:) for ln in File Lines: ??? ??? print("I do a lot of processing here") ??? ??? # Many lines of code here ..... but this loses the benefits of using "open" as a context manager, and would also be unacceptable if the file was too large to read into memory. Really I would like to write something like try: ??? with open(FileName) as f: except FileNotFoundError: ??? print(f"File {FileName} not found") ??? sys.exit() else: # or "finally:" ??? ??? for ln in f: ??? ??? ??? print("I do a lot of processing here") ??? ??? ??? # Many lines of code here ..... but this of course does not work because by the time we get to "for ln in f:" the file has been closed so we get ValueError: I/O operation on closed file I could modify the last attempt to open the file twice, which would work, but seems like a kludge (subject to race condition, inefficient). Is there a better / more Pythonic solution? Best wishes Rob Cliffe From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Jul 6 07:09:45 2024 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sat, 6 Jul 2024 07:09:45 -0400 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: On 2024-07-06 at 11:49:06 +0100, Rob Cliffe via Python-list wrote: > Is there a better / more Pythonic solution? https://docs.python.org/3/library/fileinput.html At least this attempts to abstract the problem of iterating over a file (or multiple files) into a library routine. I've used it a little, but I don't know the full depths of your use case and/or requirements. HTH, Dan From learn2program at gmail.com Sat Jul 6 07:43:40 2024 From: learn2program at gmail.com (Alan Gauld) Date: Sat, 6 Jul 2024 12:43:40 +0100 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: <05efb266-d63a-4965-8806-8733bb1ff734@yahoo.co.uk> On 06/07/2024 11:49, Rob Cliffe via Python-list wrote: > ??? If the file does not exist I want to take appropriate action, e.g. > print an error message and abort the program. > I might write it like this: > > try: > ??? with open(FileName) as f: > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > > but this violates the principle that a "try" suite should be kept small, The try is small, it only has a single statement inside. The compound block inside that statement should have its own try/ecxepts but the outer one really only applies to the with statement. I certainly prefer this option to any of the others presented. > not to mention that having "try" and "except" far apart decreases > readability. This is a valid concern although that's part of the reason we use indentation. Compared to early BASIC and FORTRAN with massive GOSUB type leaps (and often no indentation) it's very readable! But its still preferable to having the multi-level indents below or having to remember to close the file when finished (and ensure that all possible paths do so. > try: > ??? f = open(FileName) as f: > ??? FileLines = f.readlines() > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > # I forgot to put "f.close()" here -:) Exactly! That's why using with is safer even if the except is detached from the try. You are also reading the entire file into memory which could be an issue. And you are not catching any errors in the read operations because the except only covers a missing file. > Really I would like to write something like > > try: > ??? with open(FileName) as f: > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > else: # or "finally:" > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... I find that much less readable because the file handling block is a long way from the open file line and has extra control statements to mentally negotiate. The advantage of the original version is that you can ignore errors and read the code easily. You only need to find the except clause if you need to know how errors will be dealt with. But if comprehending the core functionality of the code you can just ignore all the error handling blocks. > Is there a better / more Pythonic solution? All IMHO of course, but I think the current implementation is the best of the options presented. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From oscar.j.benjamin at gmail.com Sat Jul 6 07:57:37 2024 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 6 Jul 2024 12:57:37 +0100 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: On Sat, 6 Jul 2024 at 11:55, Rob Cliffe via Python-list wrote: > > Consider this scenario (which I ran into in real life): > I want to open a text file and do a lot of processing on the lines > of that file. > If the file does not exist I want to take appropriate action, e.g. > print an error message and abort the program. > I might write it like this: > > try: > with open(FileName) as f: > for ln in f: > print("I do a lot of processing here") > # Many lines of code here ..... > except FileNotFoundError: > print(f"File {FileName} not found") > sys.exit() > > but this violates the principle that a "try" suite should be kept small, > so that only targeted exceptions are trapped, > not to mention that having "try" and "except" far apart decreases > readability. This is catching a targeted exception (FileNotFoundError) so I think it is fine. If the intention is just to call sys.exit() on error then I wouldn't worry too much about having too much code in the try. Just make sure that you do this in any other place where you open a file as well. One possible improvement is that you could catch the exception and use its filename attribute: except FileNotFoundError as e print(f"File {e.filename} not found") That way if you did catch the wrong FileNotFoundError then at least you print the correct filename. Alternatively: except FileNotFoundError as e if e.filename != FileName: raise # re-raise if not the intended exception print(f"File {e.filename} not found") For readability I would just move the many lines of code into a separate function. The reason to avoid having too much code in the try mainly applies to situations where you are going to do something other than call sys.exit() and the exception is overly generic like ValueError or TypeError. If the exception can easily be raised by a bug or something other than the intended cause then it is bad to catch exceptions around a larger block of code. If it is expected that the caller of a function might have good reason to catch the exception and handle it somehow then it is better to make a dedicated exception class and raise that instead. When there is only one place in the code that raises a particular exception type and only one place that catches it then it is usually going to be clear that you are catching the expected exception. -- Oscar From list1 at tompassin.net Sat Jul 6 09:40:49 2024 From: list1 at tompassin.net (Thomas Passin) Date: Sat, 6 Jul 2024 09:40:49 -0400 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: <95527a92-d7ec-4a97-b858-25ac03847040@tompassin.net> On 7/6/2024 6:49 AM, Rob Cliffe via Python-list wrote: > Consider this scenario (which I ran into in real life): > ??? I want to open a text file and do a lot of processing on the lines > of that file. > ??? If the file does not exist I want to take appropriate action, e.g. > print an error message and abort the program. > I might write it like this: > > try: > ??? with open(FileName) as f: > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > > but this violates the principle that a "try" suite should be kept small, > so that only targeted exceptions are trapped, > not to mention that having "try" and "except" far apart decreases > readability. > > Or I might write it like this: > > try: > ??? f = open(FileName) as f: > ??? FileLines = f.readlines() > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > # I forgot to put "f.close()" here -:) > for ln in File Lines: > ??? ??? print("I do a lot of processing here") > ??? ??? # Many lines of code here ..... > > but this loses the benefits of using "open" as a context manager, > and would also be unacceptable if the file was too large to read into > memory. > > Really I would like to write something like > > try: > ??? with open(FileName) as f: > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > else: # or "finally:" > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > > but this of course does not work because by the time we get to "for ln > in f:" the file has been closed so we get > ValueError: I/O operation on closed file > > I could modify the last attempt to open the file twice, which would > work, but seems like a kludge (subject to race condition, inefficient). > > Is there a better / more Pythonic solution? I usually read the file into a sequence of lines and then leave the open() as soon as possible. Something like this: FILENAME = 'this_is_an_example.txt' lines = None if os.path.exists(FILENAME): with open(FILENAME) as f: lines = f.readlines() # do something with lines Of course, if you want to read a huge number of lines you will need to be more thoughtful about it. Or make all the processing within the open() block be a function. Then you just have one more line in the block. From Richard at Damon-Family.org Sat Jul 6 11:01:11 2024 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 6 Jul 2024 11:01:11 -0400 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: My thoughts is that if the "many lines of code" puts the except to far from the try, then perhaps it would have made sense to factor out some part there into a function. Perhaps like: try: ?? with open(FileName) as f: ????? for ln in f{ ???????? process(ln) except FileNotFoundError: ?? print(f"File {FileName} not found:") ?? sys.exit() Now the "process" function has been factored out and can be well documented as to what it is doing on each line, and this code can be documented as running process on each line of the file. On 7/6/24 6:49 AM, Rob Cliffe via Python-list wrote: > Consider this scenario (which I ran into in real life): > ??? I want to open a text file and do a lot of processing on the lines > of that file. > ??? If the file does not exist I want to take appropriate action, e.g. > print an error message and abort the program. > I might write it like this: > > try: > ??? with open(FileName) as f: > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > > but this violates the principle that a "try" suite should be kept > small, so that only targeted exceptions are trapped, > not to mention that having "try" and "except" far apart decreases > readability. > > Or I might write it like this: > > try: > ??? f = open(FileName) as f: > ??? FileLines = f.readlines() > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > # I forgot to put "f.close()" here -:) > for ln in File Lines: > ??? ??? print("I do a lot of processing here") > ??? ??? # Many lines of code here ..... > > but this loses the benefits of using "open" as a context manager, > and would also be unacceptable if the file was too large to read into > memory. > > Really I would like to write something like > > try: > ??? with open(FileName) as f: > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > else: # or "finally:" > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > > but this of course does not work because by the time we get to "for ln > in f:" the file has been closed so we get > ValueError: I/O operation on closed file > > I could modify the last attempt to open the file twice, which would > work, but seems like a kludge (subject to race condition, inefficient). > > Is there a better / more Pythonic solution? > > Best wishes > Rob Cliffe -- Richard Damon From PythonList at DancesWithMice.info Sat Jul 6 17:05:36 2024 From: PythonList at DancesWithMice.info (dn) Date: Sun, 7 Jul 2024 09:05:36 +1200 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: On 6/07/24 22:49, Rob Cliffe via Python-list wrote: > Consider this scenario (which I ran into in real life): > ??? I want to open a text file and do a lot of processing on the lines > of that file. > ??? If the file does not exist I want to take appropriate action, e.g. > print an error message and abort the program. > I might write it like this: > > try: > ??? with open(FileName) as f: > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > > but this violates the principle that a "try" suite should be kept small, > so that only targeted exceptions are trapped, Yes! > not to mention that having "try" and "except" far apart decreases > readability. Uh-oh! - and there's a bit of a hang-over for old-timers who had to take care of file-locking within the application - we try to minimise 'time' between opening a file and closing it (etc)! As it seems the file is opened to read. Less relevant in this case, but habits and styles of coding matter... > Or I might write it like this: > > try: > ??? f = open(FileName) as f: > ??? FileLines = f.readlines() > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > # I forgot to put "f.close()" here -:) > for ln in File Lines: > ??? ??? print("I do a lot of processing here") > ??? ??? # Many lines of code here ..... > > but this loses the benefits of using "open" as a context manager, > and would also be unacceptable if the file was too large to read into > memory. So, now there are two concerns: 1 FileNotFoundError, and 2 gradual processing to avoid memory-full - added to remembering to close the file. > Really I would like to write something like > > try: > ??? with open(FileName) as f: > except FileNotFoundError: > ??? print(f"File {FileName} not found") > ??? sys.exit() > else: # or "finally:" > ??? ??? for ln in f: > ??? ??? ??? print("I do a lot of processing here") > ??? ??? ??? # Many lines of code here ..... > > but this of course does not work because by the time we get to "for ln > in f:" the file has been closed so we get > ValueError: I/O operation on closed file > > I could modify the last attempt to open the file twice, which would > work, but seems like a kludge (subject to race condition, inefficient). > > Is there a better / more Pythonic solution? Idea 1: invert the exception handling and the context-manager by writing a custom context-manager class which handles FileNotFoundError internally. Thus, calling-code becomes: with... for... processing Idea 2: incorporate idea of encapsulating "processing" into a (well-named) function to shorten the number of lines-of-code inside the with-suite. Idea 3: consider using a generator to 'produce' lines of data one-at-a-time. Remember that whilst context-managers and generators are distinct concepts within Python, they are quite similar in many ways. So, a custom generator could work like a context-manager or 'wrap' a context-manager per Idea 1. Building a custom-class (Idea 1 or Idea 3) enables the components to be kept together, per the ideal. It keeps the try-except components close and easy to relate. It is Pythonic (in the OOP style). -- Regards, =dn From cs at cskk.id.au Sat Jul 6 21:08:33 2024 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 7 Jul 2024 11:08:33 +1000 Subject: Best use of "open" context manager In-Reply-To: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: On 06Jul2024 11:49, Rob Cliffe wrote: >try: >??? f = open(FileName) as f: >??? FileLines = f.readlines() >except FileNotFoundError: >??? print(f"File {FileName} not found") >??? sys.exit() ># I forgot to put "f.close()" here -:) >for ln in File Lines: >??? ??? print("I do a lot of processing here") >??? ??? # Many lines of code here ..... What about this: try: ??? f = open(FileName) as f: except FileNotFoundError: ??? print(f"File {FileName} not found") ??? sys.exit() with f: ... process the lines here ... Remember, the `open()` call returns a file object _which can be used as a context manager_. It is separate from the `with` itself. From tomas.ukkonen at protonmail.ch Sun Jul 7 15:24:04 2024 From: tomas.ukkonen at protonmail.ch (Tomas Ukkonen) Date: Sun, 07 Jul 2024 19:24:04 +0000 Subject: PyGILState_Ensure() deadlocks, why? Message-ID: Hello Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux. /*?* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) ?* ?* g++ python_thread_test.cpp `python3-config --cflags --libs --embed` ?* ./a.out ?* ?* uname: ?* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux ?*/ #include #include #include #include void perform_long_operation() { ? ? // Simulate a long-running task ? ? std::this_thread::sleep_for(std::chrono::seconds(5)); } void thread_function() { ? ? // Ensure this thread has the GIL ? ? PyGILState_STATE gstate = PyGILState_Ensure(); ? ? // Execute some Python code ? ? PyRun_SimpleString("print('Hello from std::thread!')"); ? ? // Release the GIL for long operation ? ? Py_BEGIN_ALLOW_THREADS ? ? perform_long_operation(); ? ? Py_END_ALLOW_THREADS ? ? // Re-acquire the GIL and execute more Python code ? ? gstate = PyGILState_Ensure(); ? ? PyRun_SimpleString("print('Thread operation completed!')"); ? ? // Release the GIL ? ? PyGILState_Release(gstate); } int main() { ? ? // Initialize the Python Interpreter ? ? Py_Initialize(); ? ? // Create a vector of threads ? ? std::vector threads; ? ? // Launch threads ? ? for (int i = 0; i < 5; ++i) { ? ? ? ? threads.push_back(std::thread(thread_function)); ? ? } ? ? // Join threads ? ? for (auto& t : threads) { ? ? ? ? t.join(); ? ? } ? ? // Finalize the Python Interpreter ? ? Py_Finalize(); ? ? return 0; } Tomas Ukkonen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 509 bytes Desc: OpenPGP digital signature URL: From tomas.ukkonen at protonmail.ch Sun Jul 7 15:40:56 2024 From: tomas.ukkonen at protonmail.ch (Tomas Ukkonen) Date: Sun, 07 Jul 2024 19:40:56 +0000 Subject: PyGILState_Ensure() deadlocks, why? In-Reply-To: References: Message-ID: Hi There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++). Updated code: /*?* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) ?* ?* g++ python_thread_test.cpp `python3-config --cflags --libs --embed` ?* ./a.out ?* ?* uname: ?* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux ?*/ #include #include #include #include void thread_function() { ? ? // Ensure this thread has the GIL ? ? PyGILState_STATE gstate = PyGILState_Ensure(); ? ? // Execute some Python code ? ? PyRun_SimpleString("print('Hello from std::thread!')"); ? ? // Release the GIL ? ? PyGILState_Release(gstate); } int main() { ? ? // Initialize the Python Interpreter ? ? Py_Initialize(); ? ? // Create a vector of threads ? ? std::vector threads; ? ? // Launch threads ? ? for (int i = 0; i < 5; ++i) { ? ? ? ? threads.push_back(std::thread(thread_function)); ? ? } ? ? // Join threads ? ? for (auto& t : threads) { ? ? ? ? t.join(); ? ? } ? ? // Finalize the Python Interpreter ? ? Py_Finalize(); ? ? return 0; } sunnuntaina 7. hein?kuuta 2024 klo 10:24 ip, Tomas Ukkonen kirjoitti: > Hello > Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux. > > /*?* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) > ?* > ?* g++ python_thread_test.cpp `python3-config --cflags --libs --embed` > ?* ./a.out > ?* > ?* uname: > ?* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux > ?*/ > > #include > #include > #include > #include > > void perform_long_operation() { > ? ? // Simulate a long-running task > ? ? std::this_thread::sleep_for(std::chrono::seconds(5)); > } > > void thread_function() { > ? ? // Ensure this thread has the GIL > ? ? PyGILState_STATE gstate = PyGILState_Ensure(); > > ? ? // Execute some Python code > ? ? PyRun_SimpleString("print('Hello from std::thread!')"); > > ? ? // Release the GIL for long operation > ? ? Py_BEGIN_ALLOW_THREADS > ? ? perform_long_operation(); > ? ? Py_END_ALLOW_THREADS > > ? ? // Re-acquire the GIL and execute more Python code > ? ? gstate = PyGILState_Ensure(); > ? ? PyRun_SimpleString("print('Thread operation completed!')"); > > ? ? // Release the GIL > ? ? PyGILState_Release(gstate); > } > > int main() { > ? ? // Initialize the Python Interpreter > ? ? Py_Initialize(); > > ? ? // Create a vector of threads > ? ? std::vector threads; > > ? ? // Launch threads > ? ? for (int i = 0; i < 5; ++i) { > ? ? ? ? threads.push_back(std::thread(thread_function)); > ? ? } > > ? ? // Join threads > ? ? for (auto& t : threads) { > ? ? ? ? t.join(); > ? ? } > > ? ? // Finalize the Python Interpreter > ? ? Py_Finalize(); > > ? ? return 0; > } > > > > Tomas Ukkonen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 509 bytes Desc: OpenPGP digital signature URL: From cl at isbd.net Sat Jul 6 03:28:41 2024 From: cl at isbd.net (Chris Green) Date: Sat, 6 Jul 2024 08:28:41 +0100 Subject: Best (simplest) way to share data between processes Message-ID: <9a8nlk-jb81.ln1@q957.zbmc.eu> I have a Raspberry Pi in my boat that uses I2C to read a number of voltages and currents (using ADS1115 A2D) so I can monitor the battery condition etc. At present various different scripts (i.e. processes) just read the values using the I2C bus whenever they need to but I'm pretty sure this (quite rarely) results in false readings because two processes try to read at the same time. Thus I'm looking for ways to prevent simultaneous access. One fairly obvious way is to have single process/script which reads the A2D values continuously and writes them to a file. All other scripts then read from the file as needed, a simple file lock can then be used to prevent simultaneous access (well, simultaneous access when the writing process is writing). Is this the simplest approach? Are there better ways using multiprocess? (They look more complicated though). The I2C bus itself has a mutex but I don't think this guarantees that (for example) an A2D reading is atomic because one reading takes more than one I2C bus access. Would a mutex of some sort around each I2C transaction (i.e. complete A2D reading) be a better way to go? -- Chris Green ? From gordinator at gordinator.org Sat Jul 6 15:03:15 2024 From: gordinator at gordinator.org (Gordinator) Date: Sat, 6 Jul 2024 20:03:15 +0100 Subject: Best (simplest) way to share data between processes In-Reply-To: References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: On 06/07/2024 12:32, Stefan Ram wrote: > But why overengineer? If you feel comfortable with the file > solution, go for it! The only drawback might be that it's a > bit slower than other approaches. I absolutely agree. Overengineering is generally a bad idea because you're using a complex solution to solve a simple problem, which leads to unexpected breakage. The file solution is perfectly fine, and file locks are used by really important software (i.e package managers and such) because its simplicity makes it almost never fail. So yeah, go for it! From piergiorgio.sartor.this.should.not.be.used at nexgo.REMOVETHIS.de Sat Jul 6 15:32:44 2024 From: piergiorgio.sartor.this.should.not.be.used at nexgo.REMOVETHIS.de (Piergiorgio Sartor) Date: Sat, 6 Jul 2024 21:32:44 +0200 Subject: Best (simplest) way to share data between processes In-Reply-To: <9a8nlk-jb81.ln1@q957.zbmc.eu> References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: On 06/07/2024 09.28, Chris Green wrote: > I have a Raspberry Pi in my boat that uses I2C to read a number of > voltages and currents (using ADS1115 A2D) so I can monitor the battery > condition etc. > > At present various different scripts (i.e. processes) just read the > values using the I2C bus whenever they need to but I'm pretty sure > this (quite rarely) results in false readings because two processes > try to read at the same time. > > Thus I'm looking for ways to prevent simultaneous access. Why using "different scripts"? Is it there any particular reason? Maybe it would be better, if possible, to have a single script, which, sequentially, reads whatever needs to be read (or written). In a loop. This is even simpler than using a file. bye, pg > One fairly obvious way is to have single process/script which reads > the A2D values continuously and writes them to a file. All other > scripts then read from the file as needed, a simple file lock can then > be used to prevent simultaneous access (well, simultaneous access when > the writing process is writing). > > Is this the simplest approach? Are there better ways using > multiprocess? (They look more complicated though). > > The I2C bus itself has a mutex but I don't think this guarantees that > (for example) an A2D reading is atomic because one reading takes more > than one I2C bus access. > > Would a mutex of some sort around each I2C transaction (i.e. complete > A2D reading) be a better way to go? > -- piergiorgio From rob.cliffe at btinternet.com Sun Jul 7 17:22:49 2024 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 7 Jul 2024 22:22:49 +0100 Subject: Best use of "open" context manager In-Reply-To: References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: <1ed1c4b0-e89c-4dbc-8b16-35aeebce8ee3@btinternet.com> On 07/07/2024 02:08, Cameron Simpson wrote: > On 06Jul2024 11:49, Rob Cliffe wrote: >> try: >> ??? f = open(FileName) as f: >> ??? FileLines = f.readlines() >> except FileNotFoundError: >> ??? print(f"File {FileName} not found") >> ??? sys.exit() >> # I forgot to put "f.close()" here -:) >> for ln in File Lines: >> ??? ??? print("I do a lot of processing here") >> ??? ??? # Many lines of code here ..... > > What about this: > > ??? try: > ??? ??? f = open(FileName) as f: > ??? except FileNotFoundError: > ??? ??? print(f"File {FileName} not found") > ??? ??? sys.exit() > ??? with f: > ??????? ... process the lines here ... > > Remember, the `open()` call returns a file object _which can be used > as a context manager_. It is separate from the `with` itself. Did you test this? ??? f = open(FileName) as f: is not legal syntax. If you omit the "as f:" it's legal, but doesn't work (trying to access the file after "with f" raises the same ??? ValueError: I/O operation on closed file. I'm using Python 3.11.5. Best wishes Rob Cliffe From barry at barrys-emacs.org Sun Jul 7 18:27:01 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 7 Jul 2024 23:27:01 +0100 Subject: Best (simplest) way to share data between processes In-Reply-To: <9a8nlk-jb81.ln1@q957.zbmc.eu> References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: > On 7 Jul 2024, at 22:13, Chris Green via Python-list wrote: > > a simple file lock can then > be used to prevent simultaneous access (well, simultaneous access when > the writing process is writing). There is a simple pattern to make this robust. Write new values to a tmp file. Close the tmp file. Then use os.rename(tmpfile, productionfile). This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents, never a mix of both. Barry From python at mrabarnett.plus.com Sun Jul 7 18:47:01 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 7 Jul 2024 23:47:01 +0100 Subject: Best (simplest) way to share data between processes In-Reply-To: References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: <6e86fff1-d24c-42c9-bc72-7e22757686ec@mrabarnett.plus.com> On 2024-07-07 23:27, Barry via Python-list wrote: > > >> On 7 Jul 2024, at 22:13, Chris Green via Python-list wrote: >> >> a simple file lock can then >> be used to prevent simultaneous access (well, simultaneous access when >> the writing process is writing). > > There is a simple pattern to make this robust. > > Write new values to a tmp file. > Close the tmp file. > Then use os.rename(tmpfile, productionfile). > > This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents, never a mix of both. > For clarity I'd recommend os.replace instead. This is because on Windows os.rename it would complain if the target file already exists, but os.replace has the same behaviour on both Linux and Windows. From python at mrabarnett.plus.com Sun Jul 7 18:44:01 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 7 Jul 2024 23:44:01 +0100 Subject: PyGILState_Ensure() deadlocks, why? In-Reply-To: References: Message-ID: On 2024-07-07 20:40, Tomas Ukkonen via Python-list wrote: > Hi > > There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++). > > Updated code: > > /*?* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) > ?* > ?* g++ python_thread_test.cpp `python3-config --cflags --libs --embed` > ?* ./a.out > ?* > ?* uname: > ?* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux > ?*/ > > #include > #include > #include > #include > > > void thread_function() { > ? ? // Ensure this thread has the GIL > ? ? PyGILState_STATE gstate = PyGILState_Ensure(); > > ? ? // Execute some Python code > ? ? PyRun_SimpleString("print('Hello from std::thread!')"); > > ? ? // Release the GIL > ? ? PyGILState_Release(gstate); > } > > int main() { > ? ? // Initialize the Python Interpreter > ? ? Py_Initialize(); > > ? ? // Create a vector of threads > ? ? std::vector threads; > > ? ? // Launch threads > ? ? for (int i = 0; i < 5; ++i) { > ? ? ? ? threads.push_back(std::thread(thread_function)); > ? ? } > > ? ? // Join threads > ? ? for (auto& t : threads) { > ? ? ? ? t.join(); > ? ? } > > ? ? // Finalize the Python Interpreter > ? ? Py_Finalize(); > > ? ? return 0; > } > sunnuntaina 7. hein?kuuta 2024 klo 10:24 ip, Tomas Ukkonen kirjoitti: > >> Hello >> Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux. >> > >> /*?* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) >> ?* >> ?* g++ python_thread_test.cpp `python3-config --cflags --libs --embed` >> ?* ./a.out >> ?* >> ?* uname: >> ?* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux >> ?*/ >> > >> #include >> #include >> #include >> #include >> > >> void perform_long_operation() { >> ? ? // Simulate a long-running task >> ? ? std::this_thread::sleep_for(std::chrono::seconds(5)); >> } >> > >> void thread_function() { >> ? ? // Ensure this thread has the GIL >> ? ? PyGILState_STATE gstate = PyGILState_Ensure(); >> > >> ? ? // Execute some Python code >> ? ? PyRun_SimpleString("print('Hello from std::thread!')"); >> > >> ? ? // Release the GIL for long operation >> ? ? Py_BEGIN_ALLOW_THREADS >> ? ? perform_long_operation(); >> ? ? Py_END_ALLOW_THREADS >> > >> ? ? // Re-acquire the GIL and execute more Python code >> ? ? gstate = PyGILState_Ensure(); >> ? ? PyRun_SimpleString("print('Thread operation completed!')"); >> > >> ? ? // Release the GIL >> ? ? PyGILState_Release(gstate); >> } >> > >> int main() { >> ? ? // Initialize the Python Interpreter >> ? ? Py_Initialize(); >> At this point, there's only one thread (the main thread) and it owns the GIL. > >> ? ? // Create a vector of threads >> ? ? std::vector threads; >> > >> ? ? // Launch threads >> ? ? for (int i = 0; i < 5; ++i) { >> ? ? ? ? threads.push_back(std::thread(thread_function)); >> ? ? } >> The threads will each try to acquire and release the GIL, but it's still owned by the main thread. > >> ? ? // Join threads >> ? ? for (auto& t : threads) { >> ? ? ? ? t.join(); >> ? ? } The main thread is waiting for the sub-threads to finish, and the threads waiting for the GIL, but the main thread still owns the GIL, so they'll be waiting forever. Deadlock. >> > >> ? ? // Finalize the Python Interpreter >> ? ? Py_Finalize(); >> > >> ? ? return 0; >> } >> From barry at barrys-emacs.org Sun Jul 7 18:21:55 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 7 Jul 2024 23:21:55 +0100 Subject: PyGILState_Ensure() deadlocks, why? In-Reply-To: References: Message-ID: > On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list wrote: > > Py_Initialize(); You also need to tell python to init threading. I think you are missing more python setup code before you can use threads. Also i think you need to tell python that your thread wants to call into python. But I an not near my dev system to research this for you. I have code to use python from C++ in my pysvn project. See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp That saves the thread state and restores it. But in my case python creates the threads and I release and acquire the GIL. Barry From cs at cskk.id.au Sun Jul 7 19:02:38 2024 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 8 Jul 2024 09:02:38 +1000 Subject: Best use of "open" context manager In-Reply-To: <1ed1c4b0-e89c-4dbc-8b16-35aeebce8ee3@btinternet.com> References: <1ed1c4b0-e89c-4dbc-8b16-35aeebce8ee3@btinternet.com> Message-ID: On 07Jul2024 22:22, Rob Cliffe wrote: >>Remember, the `open()` call returns a file object _which can be used >>as a context manager_. It is separate from the `with` itself. >Did you test this? >??? f = open(FileName) as f: >is not legal syntax. No. You're right, remove the "as f:". >it's legal, but doesn't work (trying to access the file after "with f" >raises the same >??? ValueError: I/O operation on closed file. This astounds me. Code snippet to demo this? Here's a test script which I've just run now: FileName = 'foo.txt' try: f = open(FileName) except FileNotFoundError: print(f"File {FileName} not found") sys.exit() with f: for line in f: print("line:", line.rstrip()) Here's the foo.txt file: here are some lines of text Here's the run: % python3 p.py line: here are line: some lines of text From cs at cskk.id.au Sun Jul 7 22:45:21 2024 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 8 Jul 2024 12:45:21 +1000 Subject: Best use of "open" context manager In-Reply-To: <1ed1c4b0-e89c-4dbc-8b16-35aeebce8ee3@btinternet.com> References: <1ed1c4b0-e89c-4dbc-8b16-35aeebce8ee3@btinternet.com> Message-ID: On 07Jul2024 22:22, Rob Cliffe wrote: >it's legal, but doesn't work (trying to access the file after "with f" >raises the same >??? ValueError: I/O operation on closed file. Just to this: of course. The with closes the file. But my version runs the with after the try/except. From barry at barrys-emacs.org Mon Jul 8 04:34:44 2024 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 8 Jul 2024 09:34:44 +0100 Subject: Best (simplest) way to share data between processes In-Reply-To: <6e86fff1-d24c-42c9-bc72-7e22757686ec@mrabarnett.plus.com> References: <9a8nlk-jb81.ln1@q957.zbmc.eu> <6e86fff1-d24c-42c9-bc72-7e22757686ec@mrabarnett.plus.com> Message-ID: > On 7 Jul 2024, at 23:47, MRAB via Python-list wrote: > > For clarity I'd recommend os.replace instead. This is because on Windows os.rename it would complain if the target file already exists, but os.replace has the same behaviour on both Linux and Windows. Agreed. In this case the OP is on an RPi. Barry From barry at barrys-emacs.org Mon Jul 8 04:33:39 2024 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 8 Jul 2024 09:33:39 +0100 Subject: PyGILState_Ensure() deadlocks, why? In-Reply-To: References: Message-ID: <3FAD214C-6797-4A63-8F48-6E436D36A2D2@barrys-emacs.org> > On 7 Jul 2024, at 23:21, Barry via Python-list wrote: > > > >> On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list wrote: >> >> Py_Initialize(); > > You also need to tell python to init threading. I'm in front of my dev machine now and checking up on threading. There is no longer any extra init for threads required. > I think you are missing more python setup code before you can use threads. > Also i think you need to tell python that your thread wants to call into python. > But I an not near my dev system to research this for you. You are right to use PyGILState_Ensure() But as MRAB says the main thread is holding the GIL. > > I have code to use python from C++ in my pysvn project. > See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp > That saves the thread state and restores it. You still might find the classes I wrong to manage GIL acquire and release interesting. I have the C++ type system enforcing the rules of acquire and release. As well as RAII ensuring never to leave a block with the GIL in the wrong state. Barry From olegsivokon at gmail.com Sun Jul 7 17:55:09 2024 From: olegsivokon at gmail.com (Left Right) Date: Sun, 7 Jul 2024 23:55:09 +0200 Subject: Best (simplest) way to share data between processes In-Reply-To: <9a8nlk-jb81.ln1@q957.zbmc.eu> References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: If resource usage isn't an issue, then the _easy_ thing to do, that would also be easily correct is to have a server doing all the h/w-related reading and clients talking to that server. Use for the server the technology you feel most confident with. Eg. you may use Python's http package. I believe that the server from this package runs in a single thread, and thus processes all requests synchronously. So, you'll get synchronization for free. Then, the rest of the scripts that need to talk to h/w will instead be talking to this server. Again, this isn't an _efficient_ solution... but, sometimes you don't need one. And this one is easy to make, easy to debug, easy to expand. But, if instead you were looking for a more efficient solution, then, the general idea that allows the http server to work in this case would still apply: have a single synchronization program that takes requests asynchronously, and orders them. So, a basic TCP server would also work as well as a UNIX socket. Your idea with holding a lock on a file would also work (in fact, plenty of Linux utilities work that way, eg. apt-get or yum). ---- If you don't want to change the existing script, then instead of running them directly, you could run them through batch: https://man7.org/linux/man-pages/man1/batch.1p.html this is a very simply queuing program that's available for Linux. It will take care of synchronization by putting the scripts you want to run in a queue and executing them one at a time. On Sun, Jul 7, 2024 at 11:12?PM Chris Green via Python-list wrote: > > I have a Raspberry Pi in my boat that uses I2C to read a number of > voltages and currents (using ADS1115 A2D) so I can monitor the battery > condition etc. > > At present various different scripts (i.e. processes) just read the > values using the I2C bus whenever they need to but I'm pretty sure > this (quite rarely) results in false readings because two processes > try to read at the same time. > > Thus I'm looking for ways to prevent simultaneous access. > > One fairly obvious way is to have single process/script which reads > the A2D values continuously and writes them to a file. All other > scripts then read from the file as needed, a simple file lock can then > be used to prevent simultaneous access (well, simultaneous access when > the writing process is writing). > > Is this the simplest approach? Are there better ways using > multiprocess? (They look more complicated though). > > The I2C bus itself has a mutex but I don't think this guarantees that > (for example) an A2D reading is atomic because one reading takes more > than one I2C bus access. > > Would a mutex of some sort around each I2C transaction (i.e. complete > A2D reading) be a better way to go? > > -- > Chris Green > ? > -- > https://mail.python.org/mailman/listinfo/python-list From cl at isbd.net Mon Jul 8 08:56:34 2024 From: cl at isbd.net (Chris Green) Date: Mon, 8 Jul 2024 13:56:34 +0100 Subject: Best (simplest) way to share data (Posting On Python-List Prohibited) References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: <294tlk-pqr1.ln1@q957.zbmc.eu> Lawrence D'Oliveiro wrote: > On Sat, 6 Jul 2024 08:28:41 +0100, Chris Green wrote: > > > One fairly obvious way is to have single process/script which reads the > > A2D values continuously and writes them to a file. All other scripts > > then read from the file as needed, a simple file lock can then be used > > to prevent simultaneous access (well, simultaneous access when the > > writing process is writing). > > The thing with a file is, it persists even when the collector process is > not running. Do you want data that persists when the collector process is > not running? > > Is this a history of values, or just a snapshot of current values? A > history of values could be written to a database. Databases provide their > own transactions and interlocking to prevent readers from reading partial > updates. > There's a separate (crontab driven) process that writes the history to a sqlite3 database, > If it?s a snapshot of current values, that does not persist when the > collector process is not running, then why not just keep the data in the > memory of the collector process, and have it concurrently listen on a > socket for connections from readers requesting a copy of the current data? That's exactly the sort of solution I was wondering about. Is there a ready made module/library for handling this sort of thing? Basically it will just be a string of a few tens of characters that would be kept up to date by one process and asked for by all the others. -- Chris Green ? From cl at isbd.net Mon Jul 8 08:52:59 2024 From: cl at isbd.net (Chris Green) Date: Mon, 8 Jul 2024 13:52:59 +0100 Subject: Best (simplest) way to share data between processes References: <9a8nlk-jb81.ln1@q957.zbmc.eu> Message-ID: Piergiorgio Sartor wrote: > On 06/07/2024 09.28, Chris Green wrote: > > I have a Raspberry Pi in my boat that uses I2C to read a number of > > voltages and currents (using ADS1115 A2D) so I can monitor the battery > > condition etc. > > > > At present various different scripts (i.e. processes) just read the > > values using the I2C bus whenever they need to but I'm pretty sure > > this (quite rarely) results in false readings because two processes > > try to read at the same time. > > > > Thus I'm looking for ways to prevent simultaneous access. > > Why using "different scripts"? > Is it there any particular reason? > > Maybe it would be better, if possible, to have > a single script, which, sequentially, reads > whatever needs to be read (or written). > In a loop. > > This is even simpler than using a file. > Yes, but it's conceptually (and programming wise) much simpler to have separate scripts. Some of them are simple 'on demand' scripts that I run from the command line when I want to know something. Others are scripts that drive displays on control panels. -- Chris Green ? From orzodk at fastmail.com Mon Jul 8 17:09:01 2024 From: orzodk at fastmail.com (orzodk) Date: Mon, 08 Jul 2024 15:09:01 -0600 Subject: python for irc client In-Reply-To: <87msmx4mt5.fsf@raspberrypi> (Daniel via Python-list's message of "Thu, 04 Jul 2024 00:51:34 -0700") References: <87msmx4mt5.fsf@raspberrypi> Message-ID: <87zfqrfv6a.fsf@blahblah.localdomain> Daniel via Python-list writes: > One thing missing is a good textmode irc client that will connect to > quassel core. > > I've seen efforts to make a plugin for weechat but, to date, I don't see much > progress on that end. > > In your wisdom, would python be a good environment to accomplish this? I'd > likely use extended ascii and colors. The point would be to minimize the > memory footprint of the application. One feature of weechat you may be interested in is it's relay mode: https://weechat.org/files/doc/weechat/stable/weechat_user.en.html#relay If you flipped this architecture around you should be able to run weechat as your bouncer then connect quassel or any other client to weechat with it's IRC relay mode. Weechat also has a list of remote interfaces that can use this https://weechat.org/about/interfaces/ OTOH if you prefer the setup you've got I've had 0 issues with the Python plugins I've written for weechat. A Quassel/weechat proxy plugin should be pretty straight forward and if you prefer a standalone bridge, as others have suggested Python is great for that too! From rob.cliffe at btinternet.com Sat Jul 6 09:27:52 2024 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 6 Jul 2024 14:27:52 +0100 Subject: Best use of "open" context manager In-Reply-To: References: <954c4ca8-cf37-4482-a1be-46d39cb503f9@btinternet.com> Message-ID: <74a41289-d62f-465f-a5ac-665843c03528@btinternet.com> On 06/07/2024 12:57, Oscar Benjamin via Python-list wrote: > On Sat, 6 Jul 2024 at 11:55, Rob Cliffe via Python-list > wrote: >> Consider this scenario (which I ran into in real life): >> I want to open a text file and do a lot of processing on the lines >> of that file. >> If the file does not exist I want to take appropriate action, e.g. >> print an error message and abort the program. >> I might write it like this: >> >> try: >> with open(FileName) as f: >> for ln in f: >> print("I do a lot of processing here") >> # Many lines of code here ..... >> except FileNotFoundError: >> print(f"File {FileName} not found") >> sys.exit() >> >> but this violates the principle that a "try" suite should be kept small, >> so that only targeted exceptions are trapped, >> not to mention that having "try" and "except" far apart decreases >> readability. > This is catching a targeted exception (FileNotFoundError) so I think > it is fine. If the intention is just to call sys.exit() on error then > I wouldn't worry too much about having too much code in the try. Just > make sure that you do this in any other place where you open a file as > well. > > One possible improvement is that you could catch the exception and use > its filename attribute: > > except FileNotFoundError as e > print(f"File {e.filename} not found") > > That way if you did catch the wrong FileNotFoundError then at least > you print the correct filename. Good point, Oscar - thank you.? (Even if you did omit the colon on the "except" line?.? I've often thought we should have "Python without colons" as this is a mistake I frequently make.) > > Alternatively: > > except FileNotFoundError as e > if e.filename != FileName: > raise # re-raise if not the intended exception > print(f"File {e.filename} not found") Indeed, that covers all basis. > For readability I would just move the many lines of code into a > separate function. That may not always be convenient (e.g. if the many-lines-of-code needs to access a lot of local variables) but fair enough. Thanks for your answer. Rob Cliffe > > The reason to avoid having too much code in the try mainly applies to > situations where you are going to do something other than call > sys.exit() and the exception is overly generic like ValueError or > TypeError. If the exception can easily be raised by a bug or something > other than the intended cause then it is bad to catch exceptions > around a larger block of code. > > If it is expected that the caller of a function might have good reason > to catch the exception and handle it somehow then it is better to make > a dedicated exception class and raise that instead. When there is only > one place in the code that raises a particular exception type and only > one place that catches it then it is usually going to be clear that > you are catching the expected exception. > > -- > Oscar From uri at speedy.net Tue Jul 9 01:13:57 2024 From: uri at speedy.net (=?UTF-8?B?15DXldeo15k=?=) Date: Tue, 9 Jul 2024 08:13:57 +0300 Subject: A new feature request - parser add_mutually_exclusive_group - add a default value Message-ID: Hi, Please look at this Stack Overflow post: https://stackoverflow.com/questions/78722378/parser-add-mutually-exclusive-group-how-can-i-set-a-default-value 1. Is there a way to add a default to parser add_mutually_exclusive_group groups - a value that will be set by default? In this case I want test-default-languages=True to be set as a default. 2. I tried to subscribe to Python-ideas python-ideas at python.org, but I can't login to https://mail.python.org/mailman3/lists/python-ideas.python.org/. Although I did login to https://mail.python.org/mailman/options/python-list . Do I have to create a new account? Thanks, Uri. ???? uri at speedy.net From barry at barrys-emacs.org Tue Jul 9 11:40:14 2024 From: barry at barrys-emacs.org (Barry Scott) Date: Tue, 9 Jul 2024 16:40:14 +0100 Subject: A new feature request - parser add_mutually_exclusive_group - add a default value In-Reply-To: References: Message-ID: > On 9 Jul 2024, at 06:13, ????? via Python-list? wrote: > > I tried to subscribe to Python-ideas These days ideas are discussed on https://discuss.python.org/ It is rare to see an idea on the mailing list. Barry From uri at speedy.net Wed Jul 10 01:07:07 2024 From: uri at speedy.net (=?UTF-8?B?15DXldeo15k=?=) Date: Wed, 10 Jul 2024 08:07:07 +0300 Subject: A new feature request - parser add_mutually_exclusive_group - add a default value In-Reply-To: References: Message-ID: Thank you. Uri. ???? uri at speedy.net On Tue, Jul 9, 2024 at 6:40?PM Barry Scott wrote: > > > On 9 Jul 2024, at 06:13, ????? via Python-list? > wrote: > > I tried to subscribe to Python-ideas > > > These days ideas are discussed on https://discuss.python.org/ > It is rare to see an idea on the mailing list. > > Barry > > > From dpopov at anl.gov Mon Jul 8 15:09:45 2024 From: dpopov at anl.gov (Popov, Dmitry Yu) Date: Mon, 8 Jul 2024 19:09:45 +0000 Subject: Relatively prime integers in NumPy Message-ID: Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA From cl at isbd.net Tue Jul 9 06:02:39 2024 From: cl at isbd.net (Chris Green) Date: Tue, 9 Jul 2024 11:02:39 +0100 Subject: Best (simplest) way to share data References: <9a8nlk-jb81.ln1@q957.zbmc.eu> <294tlk-pqr1.ln1@q957.zbmc.eu> Message-ID: Stefan Ram wrote: > Chris Green wrote or quoted: > >That's exactly the sort of solution I was wondering about. Is there a > >ready made module/library for handling this sort of thing? Basically > >it will just be a string of a few tens of characters that would be > >kept up to date by one process and asked for by all the others. > > I'm not an expert here, and just quickly tried to make it > run, so the code will still contain errors and not contain > something necessary, but might give you a starting point. > > A process doing something (here: printing an incrementing value > named "info") and also serving requests from other processes > for this "info" value: > [snip] Thanks, that should get me started! :-) -- Chris Green ? From toby at tobiah.org Tue Jul 9 09:15:14 2024 From: toby at tobiah.org (Tobiah) Date: Tue, 9 Jul 2024 06:15:14 -0700 Subject: python repl vi mode line editing not working. Message-ID: Kubuntu 24.04. sinewave:toby ~(1)> cat .inputrc set editing-mode vi set keymap vi sinewave:toby ~(1)> cat .editrc bind -v bind \\t rl_complete sinewave:toby ~(1)> python Python 2.7.18 (default, Jul 8 2024, 12:49:12) [GCC 13.2.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> 1 1 >>> 2 2 >>> ^[k I see the literal 'escape' character + 'k', when it should let me edit previous commands. I did have to compile my own python because I'm using 2.7 on this machine. Thanks for any help. Toby From toby at tobiah.org Tue Jul 9 09:17:32 2024 From: toby at tobiah.org (Tobiah) Date: Tue, 9 Jul 2024 06:17:32 -0700 Subject: Problem using mysql library Message-ID: sinewave:toby ~(1)> python Python 2.7.18 (default, Jul 8 2024, 12:49:12) [GCC 13.2.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import MySQLdb Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 23, in (version_info, _mysql.version_info)) ImportError: this is MySQLdb version (1, 2, 5, 'final', 1), but _mysql is version (1, 4, 6, 'final', 0) I Googled this a lot, and saw many people with the same problem, but couldn't find an answer that helped. Thanks! Toby From toby at tobiah.org Wed Jul 10 09:35:16 2024 From: toby at tobiah.org (Tobiah) Date: Wed, 10 Jul 2024 06:35:16 -0700 Subject: python repl vi mode line editing not working. In-Reply-To: References: Message-ID: > For this to work, the Python implementation should use the same > readline library as your shell, I guess. It works in python3, so I guess my problem is that I'm compiling python (I think kubuntu dropped python2), but I don't see any relevant options in the configure help. From toby at tobiah.org Wed Jul 10 09:44:50 2024 From: toby at tobiah.org (Tobiah) Date: Wed, 10 Jul 2024 06:44:50 -0700 Subject: python repl vi mode line editing not working. In-Reply-To: References: Message-ID: > I see the literal 'escape' character + 'k', when it should > let me edit previous commands. > > I did have to compile my own python because I'm using 2.7 on > this machine. I figured it out. I needed to apt install libreadline-dev. From avi.e.gross at gmail.com Thu Jul 11 14:22:24 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 11 Jul 2024 14:22:24 -0400 Subject: Relatively prime integers in NumPy In-Reply-To: References: Message-ID: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://mail.python.org/mailman/listinfo/python-list From oscar.j.benjamin at gmail.com Thu Jul 11 17:22:30 2024 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 11 Jul 2024 22:22:30 +0100 Subject: Relatively prime integers in NumPy In-Reply-To: References: Message-ID: (posting on-list this time) On Thu, 11 Jul 2024 at 15:18, Popov, Dmitry Yu via Python-list wrote: > > Dear Sirs. > > Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: > [[1,5,8], > [2,4,8], > [3,3,9]] > I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. It sounds like you want the gcd (greatest common divisor) of each row. The math module can do this: In [1]: a = [[1,5,8], ...: [2,4,8], ...: [3,3,9]] In [2]: import math In [3]: [math.gcd(*row) for row in a] Out[3]: [1, 2, 3] NumPy can also do it apparently: In [10]: np.gcd.reduce(np.transpose(a)) Out[10]: array([1, 2, 3]) https://en.wikipedia.org/wiki/Greatest_common_divisor -- Oscar From avi.e.gross at gmail.com Thu Jul 11 19:26:30 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 11 Jul 2024 19:26:30 -0400 Subject: Relatively prime integers in NumPy In-Reply-To: References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> Message-ID: <014701dad3e9$c2eddc60$48c99520$@gmail.com> OK. That explains a bit more. If I understand what you are looking for is a fast implementation and quite often in Pyhon it means using code written in another language such as C that is integrated carefully in a moule or two. Another tack is to replace many explicit loops with often much faster vectorized operations. Numpy provides advantages like the above if you use it as intended. Of course there are other techniques in how code is refactored or the order of operations, or doing things in parallel. Just as an example, your inner loop ear the top is operating one at a time or numbers between 0 and max_l and hen creates variables initialized and then possibly changed in chvec and maxmult. It uses various conditions to change those variables then goes on to do more things included in a fourth nested loop. What would happen if, instead, you used two objects with the same names that were each a numpy array, or perhaps combined into a dataframe type object? Using numpy (and perhaps pandas) you could have code that initialized one such array to hold the initial 1 or 2 as needed in an object whose length was max_l+1 and then the next operations, using numpy notation would be along the lines of replace the corresponding value depending on external variables you call h or k and so on. There would be several invisible loops, perhaps chained in some way, but probably running way faster than the explicit loop. I am not going to write any specific code, but suggest you read some documentation on how to use numpy for some of the operations you want when operating on larger clusters of info. You can gain some speed even by changing a few parts. To refactor the entire thing would take more thought and if you come up with the idea of operating on a multidimensional array, might take some care. But consider what would happen if you looked at your loops which are currently of a fixed size and created a 3-D matrix with dimensions of max_h+1, max_k+1, and max_l+1 and simply initialized it with all possible initial values and then ran an algorithm to manipulate it, often asking numpy for various slices or whatever works for you as in axes. This architecture may not work for ou but is an example of the kind of thinking it an take to make a problem use algorithms more efficiently. I note the code did not actually help me understand what mathematical operation you want to perform. I assumed I might see some operations like division and t may be other parts of your code that implement what you want. But if this is a common enough need, I suspect you may want to see if something similar enough is out there. Your code may be more complex and more like the sieve of Eratosthenes that attempts to test every possibility. One algorithm I have seen simply takes the numbers you are evaluating and in a loop of the first N primes (or an open-ended generator) simply does an integer division by 2, as many times as it returns an integral result, then as many divisions by 3 then 5 and 7 and so on. It aborts when it has been chopped down to size, or the prime being used is large enough (square root or so) ad at the end, you should have some sequence of divisors, or just and the number if it is prime. Some such algorithm can be fairly fast and perhaps can even be done vectorized. One last comment is about memoization. If your data is of a nature where a relatively few numbers come up often, then you an use something, like perhaps a dictionary, to store the results of a computation like getting a list of prime factors for a specific number, or just recording whether it is prime or composite. Later calls to do calculations would always check if the result has already been saved and skip recalculating it. Good Luck From: Popov, Dmitry Yu Sent: Thursday, July 11, 2024 3:26 PM To: avi.e.gross at gmail.com; 'Popov, Dmitry Yu via Python-list' Subject: Re: Relatively prime integers in NumPy Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code _____ From: avi.e.gross at gmail.com > Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu >; 'Popov, Dmitry Yu via Python-list' > Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list > On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list > Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ From hjp-python at hjp.at Fri Jul 12 05:14:11 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 12 Jul 2024 11:14:11 +0200 Subject: Relatively prime integers in NumPy In-Reply-To: References: Message-ID: <20240712091411.tei3pcnzzy5eby6f@hjp.at> On 2024-07-08 19:09:45 +0000, Popov, Dmitry Yu via Python-list wrote: > Does NumPy provide a simple mechanism to identify relatively prime > integers, i.e. integers which don't have a common factor other than +1 > or -1? Typing "numpy gcd" into my favourite search engine brings me to https://numpy.org/doc/stable/reference/generated/numpy.gcd.html hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From sjeik_appie at hotmail.com Fri Jul 12 05:30:14 2024 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Fri, 12 Jul 2024 11:30:14 +0200 Subject: Best use of "open" context manager In-Reply-To: Message-ID: Or like below, although pylint complains about this: "consider using with". Less indentation this way. f = None try: f = open(FILENAME) records = f.readlines() except Exception: sys.exit(1) finally: if f is not None: f.close() From dpopov at anl.gov Thu Jul 11 15:25:37 2024 From: dpopov at anl.gov (Popov, Dmitry Yu) Date: Thu, 11 Jul 2024 19:25:37 +0000 Subject: Relatively prime integers in NumPy In-Reply-To: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> Message-ID: Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code ________________________________ From: avi.e.gross at gmail.com Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu ; 'Popov, Dmitry Yu via Python-list' Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ From ldo at nz.invalid Fri Jul 12 03:01:03 2024 From: ldo at nz.invalid (Lawrence D'Oliveiro) Date: Fri, 12 Jul 2024 07:01:03 -0000 (UTC) Subject: Password Hash Validation (Posting On Python-List Prohibited) References: <8734p7709v.fsf@nightsong.com> Message-ID: On Fri, 21 Jun 2024 06:32:58 -0000 (UTC), I wrote: > On Fri, 21 Jun 2024 03:40:55 -0000 (UTC), I wrote: > >> I think I will create my own wrapper using ctypes. > > Done . The repo now includes an example script that exercises the various functions of the module. From dpopov at anl.gov Fri Jul 12 13:12:59 2024 From: dpopov at anl.gov (Popov, Dmitry Yu) Date: Fri, 12 Jul 2024 17:12:59 +0000 Subject: Relatively prime integers in NumPy In-Reply-To: References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> Message-ID: Thank you very much, Oscar. Using the following code looks like a much better solution than my current Python code indeed. np.gcd.reduce(np.transpose(a)) or np.gcd.reduce(a,1) The next question is how I can generate ndarray of h,k,l indices. This can be easily done from a Python list by using the following code. import numpy as np hkl_list=[] for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????hkl_local=[] ??????????????????hkl_local.append(h) ??????????????????hkl_local.append(k) ??????????????????hkl_local.append(l) ??????????????????hkl_list.append(hkl_local) hkl=np.array(hkl_list, dtype=np.int64) This code will generate a two-dimensional ndarray of h,k,l indices but is it possible to make a faster routine with NumPy? Regards, Dmitry ________________________________ From: Python-list on behalf of Popov, Dmitry Yu via Python-list Sent: Thursday, July 11, 2024 2:25 PM To: avi.e.gross at gmail.com ; 'Popov, Dmitry Yu via Python-list' Subject: Re: Relatively prime integers in NumPy Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code ________________________________ From: avi.e.gross at gmail.com Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu ; 'Popov, Dmitry Yu via Python-list' Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!avZA_RNHnI2aBy2E2Z3kwPCY3B4aDtoxObit540PzHeIW_4s1Tkkq5NapXL3KzGXv2BTWbYQJHf6AskeTC-IEA$ From avi.e.gross at gmail.com Fri Jul 12 19:57:35 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 12 Jul 2024 19:57:35 -0400 Subject: Relatively prime integers in NumPy In-Reply-To: References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> Message-ID: <011801dad4b7$44a07100$cde15300$@gmail.com> Dmitry, I clearly did not understand what you wanted earlier as you had not made clear that in your example, you already had progressed to some level where you had the data and were now doing a second step. So, I hesitate to say much until either nobody else addressed the issue (as clearly some have) or you explain well enough. I am guessing you have programming experience in other languages and are not as ?pythonic? as some. The code you show may not be quite how others might do it. Some may write mch of your code as a single line of python using a list comprehension such as: hkl_list = [ [h, k, l] for SOMETHING in RANGE for SOMETHING2 in RANGE2 for SOMETHING3 in RANGE3] Where h, k. l come from the somethings. Back to the real world. From: Popov, Dmitry Yu Sent: Friday, July 12, 2024 1:13 PM To: avi.e.gross at gmail.com; 'Popov, Dmitry Yu via Python-list' ; oscar.j.benjamin at gmail.com; Popov, Dmitry Yu Subject: Re: Relatively prime integers in NumPy Thank you very much, Oscar. Using the following code looks like a much better solution than my current Python code indeed. np.gcd.reduce(np.transpose(a)) or np.gcd.reduce(a,1) The next question is how I can generate ndarray of h,k,l indices. This can be easily done from a Python list by using the following code. import numpy as np hkl_list=[] for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????hkl_local=[] ??????????????????hkl_local.append(h) ??????????????????hkl_local.append(k) ??????????????????hkl_local.append(l) ??????????????????hkl_list.append(hkl_local) hkl=np.array(hkl_list, dtype=np.int64) This code will generate a two-dimensional ndarray of h,k,l indices but is it possible to make a faster routine with NumPy? Regards, Dmitry _____ From: Python-list > on behalf of Popov, Dmitry Yu via Python-list > Sent: Thursday, July 11, 2024 2:25 PM To: avi.e.gross at gmail.com >; 'Popov, Dmitry Yu via Python-list' > Subject: Re: Relatively prime integers in NumPy Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code ________________________________ From: avi.e.gross at gmail.com > Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu >; 'Popov, Dmitry Yu via Python-list' > Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list > On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list > Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!avZA_RNHnI2aBy2E2Z3kwPCY3B4aDtoxObit540PzHeIW_4s1Tkkq5NapXL3KzGXv2BTWbYQJHf6AskeTC-IEA$ From dpopov at anl.gov Fri Jul 12 23:10:12 2024 From: dpopov at anl.gov (Popov, Dmitry Yu) Date: Sat, 13 Jul 2024 03:10:12 +0000 Subject: Relatively prime integers in NumPy In-Reply-To: <011801dad4b7$44a07100$cde15300$@gmail.com> References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> <011801dad4b7$44a07100$cde15300$@gmail.com> Message-ID: Thank you very much. List comprehensions make code much more concise indeed. Do list comprehensions also improve the speed of calculations? ________________________________ From: avi.e.gross at gmail.com Sent: Friday, July 12, 2024 6:57 PM To: Popov, Dmitry Yu ; 'Popov, Dmitry Yu via Python-list' ; oscar.j.benjamin at gmail.com Subject: RE: Relatively prime integers in NumPy Dmitry, I clearly did not understand what you wanted earlier as you had not made clear that in your example, you already had progressed to some level where you had the data and were now doing a second step. So, I hesitate to say much until ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dmitry, I clearly did not understand what you wanted earlier as you had not made clear that in your example, you already had progressed to some level where you had the data and were now doing a second step. So, I hesitate to say much until either nobody else addressed the issue (as clearly some have) or you explain well enough. I am guessing you have programming experience in other languages and are not as ?pythonic? as some. The code you show may not be quite how others might do it. Some may write mch of your code as a single line of python using a list comprehension such as: hkl_list = [ [h, k, l] for SOMETHING in RANGE for SOMETHING2 in RANGE2 for SOMETHING3 in RANGE3] Where h, k. l come from the somethings. Back to the real world. From: Popov, Dmitry Yu Sent: Friday, July 12, 2024 1:13 PM To: avi.e.gross at gmail.com; 'Popov, Dmitry Yu via Python-list' ; oscar.j.benjamin at gmail.com; Popov, Dmitry Yu Subject: Re: Relatively prime integers in NumPy Thank you very much, Oscar. Using the following code looks like a much better solution than my current Python code indeed. np.gcd.reduce(np.transpose(a)) or np.gcd.reduce(a,1) The next question is how I can generate ndarray of h,k,l indices. This can be easily done from a Python list by using the following code. import numpy as np hkl_list=[] for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????hkl_local=[] ??????????????????hkl_local.append(h) ??????????????????hkl_local.append(k) ??????????????????hkl_local.append(l) ??????????????????hkl_list.append(hkl_local) hkl=np.array(hkl_list, dtype=np.int64) This code will generate a two-dimensional ndarray of h,k,l indices but is it possible to make a faster routine with NumPy? Regards, Dmitry ________________________________ From: Python-list > on behalf of Popov, Dmitry Yu via Python-list > Sent: Thursday, July 11, 2024 2:25 PM To: avi.e.gross at gmail.com >; 'Popov, Dmitry Yu via Python-list' > Subject: Re: Relatively prime integers in NumPy Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code ________________________________ From: avi.e.gross at gmail.com > Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu >; 'Popov, Dmitry Yu via Python-list' > Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list > On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list > Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!avZA_RNHnI2aBy2E2Z3kwPCY3B4aDtoxObit540PzHeIW_4s1Tkkq5NapXL3KzGXv2BTWbYQJHf6AskeTC-IEA$ From avi.e.gross at gmail.com Sat Jul 13 00:46:55 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 13 Jul 2024 00:46:55 -0400 Subject: Relatively prime integers in NumPy In-Reply-To: References: <00e801dad3bf$473daed0$d5b90c70$@gmail.com> <011801dad4b7$44a07100$cde15300$@gmail.com> Message-ID: <01ef01dad4df$aff92ff0$0feb8fd0$@gmail.com> Dmitry, Efficiency of several kinds is hotly debated and sometimes it depends a lot on what is done within loops. Many suggest a mild speed up of some comprehensions over loops but the loops are not gone but somewhat hidden and perhaps some aspects are faster for having been written in C carefully and not interpreted. Comprehensions (and there are other versions that generate dictionaries and tuples and sets) may also be sped up a bit for other reasons like your fairly expensive APPPEND that has to keep finding the end o f a growing list and is not done the same way in a comprehension. If you do a search, you find many opinions including on using functional programming techniques such as map/reduce. There are also Your particular case is interesting because it just makes all combination of three variables. Some languages, like R, have functions that do this for you, like expand.grd. Python has many modules, like itertools that do things including combinations but perhaps not designed for your case. Here is a version of your scenario: import itertools a = range(3) b = range(4) c = range(5) list(itertools.product(a,b,c)) The result comes as tuples but as you are moving the result into numpy, does it matter: >>> list(itertools.product(a,b,c)) [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 1, 0), (0, 1, 1), (0, 1, 2), (0, 1, 3), (0, 1, 4), (0, 2, 0), (0, 2, 1), (0, 2, 2), (0, 2, 3), (0, 2, 4), (0, 3, 0), (0, 3, 1), (0, 3, 2), (0, 3, 3), (0, 3, 4), (1, 0, 0), (1, 0, 1), (1, 0, 2), (1, 0, 3), (1, 0, 4), (1, 1, 0), (1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 1, 4), (1, 2, 0), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 2, 4), (1, 3, 0), (1, 3, 1), (1, 3, 2), (1, 3, 3), (1, 3, 4), (2, 0, 0), (2, 0, 1), (2, 0, 2), (2, 0, 3), (2, 0, 4), (2, 1, 0), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 1, 4), (2, 2, 0), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 2, 4), (2, 3, 0), (2, 3, 1), (2, 3, 2), (2, 3, 3), (2, 3, 4)] Or a atd easier to read pretty printed: >>> import pprint >>> pprint.pprint(list(itertools.product(a,b,c))) [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 1, 0), (0, 1, 1), (0, 1, 2), (0, 1, 3), (0, 1, 4), (0, 2, 0), (0, 2, 1), (0, 2, 2), (0, 2, 3), (0, 2, 4), (0, 3, 0), (0, 3, 1), (0, 3, 2), (0, 3, 3), (0, 3, 4), (1, 0, 0), (1, 0, 1), (1, 0, 2), (1, 0, 3), (1, 0, 4), (1, 1, 0), (1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 1, 4), (1, 2, 0), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 2, 4), (1, 3, 0), (1, 3, 1), (1, 3, 2), (1, 3, 3), (1, 3, 4), (2, 0, 0), (2, 0, 1), (2, 0, 2), (2, 0, 3), (2, 0, 4), (2, 1, 0), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 1, 4), (2, 2, 0), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 2, 4), (2, 3, 0), (2, 3, 1), (2, 3, 2), (2, 3, 3), (2, 3, 4)] I think that is close enough to what you want but is it faster? You can try a benchmarking method on alternatives. From: Popov, Dmitry Yu Sent: Friday, July 12, 2024 11:10 PM To: avi.e.gross at gmail.com; 'Popov, Dmitry Yu via Python-list' ; oscar.j.benjamin at gmail.com Subject: Re: Relatively prime integers in NumPy Thank you very much. List comprehensions make code much more concise indeed. Do list comprehensions also improve the speed of calculations? _____ From: avi.e.gross at gmail.com > Sent: Friday, July 12, 2024 6:57 PM To: Popov, Dmitry Yu >; 'Popov, Dmitry Yu via Python-list' >; oscar.j.benjamin at gmail.com > Subject: RE: Relatively prime integers in NumPy Dmitry, I clearly did not understand what you wanted earlier as you had not made clear that in your example, you already had progressed to some level where you had the data and were now doing a second step. So, I hesitate to say much until ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dmitry, I clearly did not understand what you wanted earlier as you had not made clear that in your example, you already had progressed to some level where you had the data and were now doing a second step. So, I hesitate to say much until either nobody else addressed the issue (as clearly some have) or you explain well enough. Ditr I am guessing you have programming experience in other languages and are not as ?pythonic? as some. The code you show may not be quite how others might do it. Some may write mch of your code as a single line of python using a list comprehension such as: hkl_list = [ [h, k, l] for SOMETHING in RANGE for SOMETHING2 in RANGE2 for SOMETHING3 in RANGE3] Where h, k. l come from the somethings. Back to the real world. From: Popov, Dmitry Yu > Sent: Friday, July 12, 2024 1:13 PM To: avi.e.gross at gmail.com ; 'Popov, Dmitry Yu via Python-list' >; oscar.j.benjamin at gmail.com ; Popov, Dmitry Yu > Subject: Re: Relatively prime integers in NumPy Thank you very much, Oscar. Using the following code looks like a much better solution than my current Python code indeed. np.gcd.reduce(np.transpose(a)) or np.gcd.reduce(a,1) The next question is how I can generate ndarray of h,k,l indices. This can be easily done from a Python list by using the following code. import numpy as np hkl_list=[] for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????hkl_local=[] ??????????????????hkl_local.append(h) ??????????????????hkl_local.append(k) ??????????????????hkl_local.append(l) ??????????????????hkl_list.append(hkl_local) hkl=np.array(hkl_list, dtype=np.int64) This code will generate a two-dimensional ndarray of h,k,l indices but is it possible to make a faster routine with NumPy? Regards, Dmitry _____ From: Python-list > on behalf of Popov, Dmitry Yu via Python-list > Sent: Thursday, July 11, 2024 2:25 PM To: avi.e.gross at gmail.com >; 'Popov, Dmitry Yu via Python-list' > Subject: Re: Relatively prime integers in NumPy Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for your interest. My explanation is too concise indeed, sorry. So far, I have used Python code with three enclosed 'for' loops for this purpose which is pretty time consuming. I'm trying to develop a NumPy based code to make this procedure faster. This routine is kind of 'heart' of the algorithm to index of X-ray Laue diffraction patterns. In our group we have to process huge amount of such patterns. They are collected at a synchrotron radiation facility. Faster indexation routine would help a lot. This is the code I'm currently using. Any prompts how to implement it in NumPy would be highly appreciated. for h in range(0, max_h): ??????for k in range(0, max_k): ????????????for l in range(0, max_l): ??????????????????chvec=1 ??????????????????maxmult=2 ??????????????????if h > 1:????????????????????? ????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????maxmult=l ??????????????????if h > 1: ????????????????????????if maxmult > h: ??????????????????????????????maxmult=h ??????????????????if k > 1: ????????????????????????if maxmult > k: ??????????????????????????????maxmult=k ??????????????????if l > 1: ????????????????????????if maxmult > l: ??????????????????????????????maxmult=l ??????????????????maxmult=maxmult+1 ??????????????????for innen in range(2, maxmult): ????????????????????????if h in range(0, (max_h+1), innen): ??????????????????????????????if k in range(0, (max_k+1), innen): ????????????????????????????????????if l in range(0, (max_l+1), innen): ??????????????????????????????????????????chvec=0 ??????????????????if chvec==1: ????????????????????????# Only relatively prime integers h,k,l pass to this block of the code ________________________________ From: avi.e.gross at gmail.com > Sent: Thursday, July 11, 2024 1:22 PM To: Popov, Dmitry Yu >; 'Popov, Dmitry Yu via Python-list' > Subject: RE: Relatively prime integers in NumPy ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd ???????, You may think you explained what you wanted but I do not see what result you expect from your examples. Your request is a bit too esoteric to be a great candidate for being built into a module like numpy for general purpose se but I can imagine it could be available in modules build on top of numpy. Is there a reason you cannot solve this mostly outside numpy? It looks like you could use numpy to select the numbers you want to compare, then call one of many methods you can easily search for to see how to use python to make some list or other data structure for divisors of each number involved and then use standard methods to compare the lists and exact common divisors. If needed, you could then put the results back into your original data structure using numpy albeit the number of matches can vary. Maybe a better explanation is needed as I cannot see what your latter words about -1 and 1 are about. Perhaps someone else knows. -----Original Message----- From: Python-list > On Behalf Of Popov, Dmitry Yu via Python-list Sent: Monday, July 8, 2024 3:10 PM To: Popov, Dmitry Yu via Python-list > Subject: Relatively prime integers in NumPy Dear Sirs. Does NumPy provide a simple mechanism to identify relatively prime integers, i.e. integers which don't have a common factor other than +1 or -1? For example, in case of this array: [[1,5,8], [2,4,8], [3,3,9]] I can imagine a function which would return array of common factors along axis 0: [1,2,3]. Those triples of numbers along axis 1 with the factor of1 or -1 would be relatively prime integers. Regards, Dmitry Popov Argonne, IL USA -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!ZGK1ZXYgmC6cpNa1xTXVTNklhunjYiinwaDe_xE3sJyVs4ZcVgUB_v2FKvDzDspx7IzFCZI7JpFsiV5iH58P$ -- https://urldefense.us/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!G_uCfscf7eWS!avZA_RNHnI2aBy2E2Z3kwPCY3B4aDtoxObit540PzHeIW_4s1Tkkq5NapXL3KzGXv2BTWbYQJHf6AskeTC-IEA$ From thomas at python.org Thu Jul 18 10:20:09 2024 From: thomas at python.org (Thomas Wouters) Date: Thu, 18 Jul 2024 16:20:09 +0200 Subject: [RELEASE] Python 3.13.0 beta 4 released. Message-ID: Python 3.13.0b4, the final beta of Python 3.13, is now available: https://www.python.org/downloads/release/python-3130b4/ *This is a beta preview of Python 3.13* Python 3.13 is still in development. This release, 3.13.0b4, is the *final* beta release preview of 3.13. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We *strongly encourage* maintainers of third-party Python projects to *test with 3.13* during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2024-07-30). Our goal is to have *no ABI changes* after this final beta release, and as few code changes as possible after 3.13.0rc1, the first release candidate. To achieve that, it will be *extremely important* to get as much exposure for 3.13 as possible during the beta phase. Please keep in mind that this is a preview release and its use is *not* recommended for production environments. Major new features of the 3.13 series, compared to 3.12 Some of the new major new features and changes in Python 3.13 are: New features - A new and improved interactive interpreter , based on PyPy ?s, featuring multi-line editing and color support, as well as colorized exception tracebacks . - An *experimental* free-threaded build mode , which disables the Global Interpreter Lock, allowing threads to run more concurrently. The build mode is available as an experimental feature in the Windows and macOS installers as well. - A preliminary, *experimental* JIT , providing the ground work for significant performance improvements. - The locals() builtin function (and its C equivalent) now has well-defined semantics when mutating the returned mapping , which allows debuggers to operate more consistently. - The (cyclic) garbage collector is now incremental , which should mean shorter pauses for collection in programs with a lot of objects. - A modified version of mimalloc is now included, optional but enabled by default if supported by the platform, and required for the free-threaded build mode. - Docstrings now have their leading indentation stripped , reducing memory use and the size of .pyc files. (Most tools handling docstrings already strip leading indentation.) - The dbm module has a new dbm.sqlite3 backend that is used by default when creating new files. - The minimum supported macOS version was changed from 10.9 to *10.13 (High Sierra)*. Older macOS versions will not be supported going forward. - WASI is now a Tier 2 supported platform . Emscripten is no longer an officially supported platform (but Pyodide continues to support Emscripten). Typing - Support for type defaults in type parameters . - A new type narrowing annotation , typing.TypeIs. - A new annotation for read-only items in TypeDicts . - A new annotation for marking deprecations in the type system . Removals and new deprecations - PEP 594 (Removing dead batteries from the standard library) scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3. - Many other removals of deprecated classes, functions and methods in various standard library modules. - C API removals and deprecations . (Some removals present in alpha 1 were reverted in alpha 2, as the removals were deemed too disruptive at this time.) - New deprecations , most of which are scheduled for removal from Python 3.15 or 3.16. (Hey, *fellow core developer,* if a feature you find important is missing from this list, let Thomas know .) For more details on the changes to Python 3.13, see What?s new in Python 3.13 . The next pre-release of Python 3.13 will be 3.13.0rc1, *the first release candidate*, currently scheduled for 2024-07-30. More resources - Online Documentation - PEP 719 , 3.13 Release Schedule - Report bugs at https://github.com/python/cpython/issues. - Help fund Python directly (or via GitHub Sponsors ), and support the Python community . Enjoy the new releases Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Finally, regards from a confusingly hot and rainy Munich, Your release team, Thomas Wouters ?ukasz Langa Ned Deily Steve Dower From liznashah40 at gmail.com Fri Jul 26 07:25:48 2024 From: liznashah40 at gmail.com (Lizna Shah) Date: Fri, 26 Jul 2024 17:10:48 +0545 Subject: Issue with pip Installation on My Laptop Message-ID: Hello, I am experiencing a problem with pip not being installed on my laptop and would appreciate any assistance you can provide. Here are the details of my issue: - Operating System: Windows Python Version: Python 3.10.10 Steps I have already taken to try and resolve the issue: 1. Verified that Python is installed by running `python --version` (output: [Python 3.10.10]). 2. Tried to install pip using the command `python -m ensurepip` and received the following error: Traceback (most recent call last): File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\ensurepip\__main__.py", line 5, in sys.exit(ensurepip._main()) File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\ensurepip\__init__.py", line 287, in _main return _bootstrap( File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\ensurepip\__init__.py", line 203, in _bootstrap return _run_pip([*args, *_PACKAGE_NAMES], additional_paths) File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\ensurepip\__init__.py", line 104, in _run_pip return subprocess.run(cmd, check=True).returncode File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 503, in run with Popen(*popenargs, **kwargs) as process: File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 971, in init self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Users\LIZNA SHAH\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1440, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, OSError: [WinError 225] Operation did not complete successfully because the file contains a virus or potentially unwanted software 3. Attempted to use `get-pip.py` script from the official website, but encountered: ['get-pip.py' is not recognized as an internal or external command, operable program or batch file.]. Additional information: - I was able to use pip successfully with previous versions of Python on this laptop. I would be grateful for any guidance or troubleshooting steps you could suggest to resolve this issue. Thank you. Best regards, Lizna Shah Nepal From list1 at tompassin.net Fri Jul 26 18:28:00 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 26 Jul 2024 18:28:00 -0400 Subject: Issue with pip Installation on My Laptop In-Reply-To: References: Message-ID: <199a0fc1-b586-4988-8964-35d5c8a49b2a@tompassin.net> On 7/26/2024 7:25 AM, Lizna Shah via Python-list wrote: > OSError: [WinError 225] Operation did not complete successfully because the > file contains a virus or potentially unwanted software That part of the error message tells you the story. Windows thinks some file in the install has been corrupted with malware. From mats at wichmann.us Sat Jul 27 16:58:08 2024 From: mats at wichmann.us (Mats Wichmann) Date: Sat, 27 Jul 2024 14:58:08 -0600 Subject: Issue with pip Installation on My Laptop In-Reply-To: <199a0fc1-b586-4988-8964-35d5c8a49b2a@tompassin.net> References: <199a0fc1-b586-4988-8964-35d5c8a49b2a@tompassin.net> Message-ID: <060e6f6f-6cd4-4c0a-809c-a3af949083ab@wichmann.us> On 7/26/24 16:28, Thomas Passin via Python-list wrote: > On 7/26/2024 7:25 AM, Lizna Shah via Python-list wrote: >> OSError: [WinError 225] Operation did not complete successfully >> because the >> file contains a virus or potentially unwanted software > > That part of the error message tells you the story.? Windows thinks some > file in the install has been corrupted with malware. > The Windows installer comes with pip, there's no need to do an extra install to get it: python -m pip --version If you can't find the pip *command*, that's a problem with yout PATH settings. The Python installer offers to add the location of Python itself to PATH, and you've apparently taken it up on that offer, but that's not the same directory that pip goes to. Just use it as a module and you should be fine. === The typical paths will be something like C:\Users\you\AppData\Local\Programs\Python\Python310 # python executable C:\Users\you\AppData\Local\Programs\Python\Python310\Scripts # pip "executable" From python at mrabarnett.plus.com Sat Jul 27 19:13:10 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 28 Jul 2024 00:13:10 +0100 Subject: Issue with pip Installation on My Laptop In-Reply-To: <060e6f6f-6cd4-4c0a-809c-a3af949083ab@wichmann.us> References: <199a0fc1-b586-4988-8964-35d5c8a49b2a@tompassin.net> <060e6f6f-6cd4-4c0a-809c-a3af949083ab@wichmann.us> Message-ID: On 2024-07-27 21:58, Mats Wichmann via Python-list wrote: > On 7/26/24 16:28, Thomas Passin via Python-list wrote: >> On 7/26/2024 7:25 AM, Lizna Shah via Python-list wrote: >>> OSError: [WinError 225] Operation did not complete successfully >>> because the >>> file contains a virus or potentially unwanted software >> >> That part of the error message tells you the story.? Windows thinks some >> file in the install has been corrupted with malware. >> > > > The Windows installer comes with pip, there's no need to do an extra > install to get it: > > python -m pip --version > On Windows it's recommended that you use the Python Launcher 'py': py -m pip --version > If you can't find the pip *command*, that's a problem with yout PATH > settings. The Python installer offers to add the location of Python > itself to PATH, and you've apparently taken it up on that offer, but > that's not the same directory that pip goes to. Just use it as a module > and you should be fine. > > > === > > The typical paths will be something like > > C:\Users\you\AppData\Local\Programs\Python\Python310 # python executable > C:\Users\you\AppData\Local\Programs\Python\Python310\Scripts # pip > "executable" From mats at wichmann.us Sat Jul 27 20:10:41 2024 From: mats at wichmann.us (Mats Wichmann) Date: Sat, 27 Jul 2024 18:10:41 -0600 Subject: Issue with pip Installation on My Laptop In-Reply-To: References: <199a0fc1-b586-4988-8964-35d5c8a49b2a@tompassin.net> <060e6f6f-6cd4-4c0a-809c-a3af949083ab@wichmann.us> Message-ID: On 7/27/24 17:13, MRAB via Python-list wrote: > On 2024-07-27 21:58, Mats Wichmann via Python-list wrote: >> On 7/26/24 16:28, Thomas Passin via Python-list wrote: >>> On 7/26/2024 7:25 AM, Lizna Shah via Python-list wrote: >>>> OSError: [WinError 225] Operation did not complete successfully >>>> because the >>>> file contains a virus or potentially unwanted software >>> >>> That part of the error message tells you the story.? Windows thinks >>> some file in the install has been corrupted with malware. >>> >> >> >> The Windows installer comes with pip, there's no need to do an extra >> install to get it: >> >> python -m pip --version >> > On Windows it's recommended that you use the Python Launcher 'py': > > py -m pip --version I agree! :-) but since the OP had apparently done enough to get the command named "python" to work, was trying to not introduce one extra factor. From iamybj at icloud.com Sat Jul 27 06:36:17 2024 From: iamybj at icloud.com (Benjamin) Date: Sat, 27 Jul 2024 18:36:17 +0800 Subject: I need a free LAMP hosting and a iMac Message-ID: Hi, For several months I have searched free web hosing in Google, but have not find a satisfying result now. Any body know some good LAMP free web hosting? And, I have lost job since 2018, my macbook has only 2 intel core, I want to buy a new iMac for person programming, but I have only little money, any body donate me a iMac with ARM CPU? or donate some money at https://buymeacoffee.com/benjamin_yin Thanks. From sch at fedora.email Tue Jul 30 13:36:43 2024 From: sch at fedora.email (Schimon Jehudah) Date: Tue, 30 Jul 2024 20:36:43 +0300 Subject: Installation of Slixfeed with pip fails Message-ID: <20240730203643.7810f6dc@workstation.localdomain> Greetings, to one and all! My name is Schimon, and I am the developer of a news chat bot for the XMPP network, called Slixfeed. I have recently added support for OMEMO encryption, and a friend of mine has reported that there is an issue installing it with pip. I suppoes this is a fault of a package at PyPi, or a fault at my pyproject.toml. This is the link to the project: https://git.xmpp-it.net/sch/Slixfeed#getting-started Please advise, Schimon From mk1853387 at gmail.com Tue Jul 30 14:18:42 2024 From: mk1853387 at gmail.com (marc nicole) Date: Tue, 30 Jul 2024 20:18:42 +0200 Subject: Predicting an object over an pretrained model is not working Message-ID: Hello all, I want to predict an object by given as input an image and want to have my model be able to predict the label. I have trained a model using tensorflow based on annotated database where the target object to predict was added to the pretrained model. the code I am using is the following where I set the target object image as input and want to have the prediction output: class MultiObjectDetection(): def __init__(self, classes_name): self._classes_name = classes_name self._num_classes = len(classes_name) self._common_params = {'image_size': 448, 'num_classes': self._num_classes, 'batch_size':1} self._net_params = {'cell_size': 7, 'boxes_per_cell':2, 'weight_decay': 0.0005} self._net = YoloTinyNet(self._common_params, self._net_params, test=True) def predict_object(self, image): predicts = self._net.inference(image) return predicts def process_predicts(self, resized_img, predicts, thresh=0.2): """ process the predicts of object detection with one image input. Args: resized_img: resized source image. predicts: output of the model. thresh: thresh of bounding box confidence. Return: predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], [...]]}. """ cls_num = self._num_classes bbx_per_cell = self._net_params["boxes_per_cell"] cell_size = self._net_params["cell_size"] img_size = self._common_params["image_size"] p_classes = predicts[0, :, :, 0:cls_num] C = predicts[0, :, :, cls_num:cls_num+bbx_per_cell] # two bounding boxes in one cell. coordinate = predicts[0, :, :, cls_num+bbx_per_cell:] # all bounding boxes position. p_classes = np.reshape(p_classes, (cell_size, cell_size, 1, cls_num)) C = np.reshape(C, (cell_size, cell_size, bbx_per_cell, 1)) P = C * p_classes # confidencefor all classes of all bounding boxes (cell_size, cell_size, bounding_box_num, class_num) = (7, 7, 2, 1). predicts_dict = {} for i in range(cell_size): for j in range(cell_size): temp_data = np.zeros_like(P, np.float32) temp_data[i, j, :, :] = P[i, j, :, :] position = np.argmax(temp_data) # refer to the class num (with maximum confidence) for every bounding box. index = np.unravel_index(position, P.shape) if P[index] > thresh: class_num = index[-1] coordinate = np.reshape(coordinate, (cell_size, cell_size, bbx_per_cell, 4)) # (cell_size, cell_size, bbox_num_per_cell, coordinate)[xmin, ymin, xmax, ymax] max_coordinate = coordinate[index[0], index[1], index[2], :] xcenter = max_coordinate[0] ycenter = max_coordinate[1] w = max_coordinate[2] h = max_coordinate[3] xcenter = (index[1] + xcenter) * (1.0*img_size /cell_size) ycenter = (index[0] + ycenter) * (1.0*img_size /cell_size) w = w * img_size h = h * img_size xmin = 0 if (xcenter - w/2.0 < 0) else (xcenter - w/2.0) ymin = 0 if (xcenter - w/2.0 < 0) else (ycenter - h/2.0) xmax = resized_img.shape[0] if (xmin + w) > resized_img.shape[0] else (xmin + w) ymax = resized_img.shape[1] if (ymin + h) > resized_img.shape[1] else (ymin + h) class_name = self._classes_name[class_num] predicts_dict.setdefault(class_name, []) predicts_dict[class_name].append([int(xmin), int(ymin), int(xmax), int(ymax), P[index]]) return predicts_dict def non_max_suppress(self, predicts_dict, threshold=0.5): """ implement non-maximum supression on predict bounding boxes. Args: predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], [...]]}. threshhold: iou threshold Return: predicts_dict processed by non-maximum suppression """ for object_name, bbox in predicts_dict.items(): bbox_array = np.array(bbox, dtype=np.float) x1, y1, x2, y2, scores = bbox_array[:,0], bbox_array[:,1], bbox_array[:,2], bbox_array[:,3], bbox_array[:,4] areas = (x2-x1+1) * (y2-y1+1) order = scores.argsort()[::-1] keep = [] while order.size > 0: i = order[0] keep.append(i) xx1 = np.maximum(x1[i], x1[order[1:]]) yy1 = np.maximum(y1[i], y1[order[1:]]) xx2 = np.minimum(x2[i], x2[order[1:]]) yy2 = np.minimum(y2[i], y2[order[1:]]) inter = np.maximum(0.0, xx2-xx1+1) * np.maximum(0.0, yy2-yy1+1) iou = inter/(areas[i]+areas[order[1:]]-inter) indexs = np.where(iou<=threshold)[0] order = order[indexs+1] bbox = bbox_array[keep] predicts_dict[object_name] = bbox.tolist() predicts_dict = predicts_dict return predicts_dict class_names = ["aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor", "small_ball"] modelFile = ('models\\train\\model.ckpt-0') track_object = "small_ball"print("object detection and tracking...") multiObjectDetect = MultiObjectDetection(IP, class_names) image = tf.placeholder(tf.float32, (1, 448, 448, 3)) object_predicts = multiObjectDetect.predict_object(image) sess = tf.Session() saver = tf.train.Saver(multiObjectDetect._net.trainable_collection) saver.restore(sess, modelFile) index = 0while 1: src_img = cv2.imread("./weirdobject.jpg") resized_img = cv2.resize(src_img, (448, 448)) np_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2RGB) np_img = np_img.astype(np.float32) np_img = np_img / 255.0 * 2 - 1 np_img = np.reshape(np_img, (1, 448, 448, 3)) np_predict = sess.run(object_predicts, feed_dict={image: np_img}) predicts_dict = multiObjectDetect.process_predicts(resized_img, np_predict) predicts_dict = multiObjectDetect.non_max_suppress(predicts_dict) print ("predict dict = ", predicts_dict) The problem with this code is that the predicts_dict returns: predict dict = {'sheep': [[233.0, 92.0, 448.0, -103.0, 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], [0.0, 0.0, 448.0, 431.0, 3.393721580505371]], 'horse': [[374.0, 33.0, 282.0, 448.0, 5.277851581573486], [135.0, 688.0, -33.0, -14.0, 3.5144259929656982], [1.0, 117.0, 112.0, -138.0, 2.656987190246582]], 'bicycle': [[461.0, 781.0, 154.0, -381.0, 5.918102741241455], [70.0, 344.0, 391.0, -138.0, 3.031444787979126], [378.0, 497.0, 46.0, 149.0, 2.7629122734069824], [541.0, 583.0, 69.0, 307.0, 2.7170517444610596], [323.0, 22.0, 336.0, 448.0, 1.608760952949524]], 'bottle': [[390.0, 218.0, -199.0, 448.0, 4.582971096038818], [0.0, 0.0, 448.0, -410.0, 0.9097045063972473]], 'sofa': [[346.0, 102.0, 323.0, -38.0, 2.371835947036743]], 'dog': [[319.0, 254.0, -282.0, 373.0, 4.022889137268066]], 'cat': [[63.0, -195.0, 365.0, -92.0, 3.5134828090667725]], 'person': [[22.0, -122.0, 154.0, 448.0, 3.927537441253662], [350.0, 155.0, -36.0, -445.0, 2.679833173751831], [119.0, 416.0, -43.0, 292.0, 0.9529445171356201], [251.0, 445.0, 225.0, 188.0, 0.9001350402832031]], 'train': [[329.0, 485.0, -24.0, -235.0, 2.7050414085388184], [483.0, 362.0, 237.0, -86.0, 2.555817127227783], [13.0, 365.0, 373.0, 448.0, 0.6229299902915955]], 'small_ball': [[217.0, 737.0, 448.0, -315.0, 1.739920973777771], [117.0, 283.0, 153.0, 122.0, 1.5690066814422607]], 'boat': [[164.0, 805.0, 34.0, -169.0, 4.972668170928955], [0.0, 0.0, 397.0, 69.0, 2.353729486465454], [302.0, 605.0, 15.0, -22.0, 2.0259625911712646]], 'aeroplane': [[470.0, 616.0, -305.0, -37.0, 3.431873321533203], [0.0, 0.0, 448.0, -72.0, 2.836672306060791]], 'bus': [[0.0, 0.0, -101.0, -280.0, 1.2078320980072021]], 'pottedplant': [[620.0, -268.0, -124.0, 418.0, 2.158564805984497], [0.0, 0.0, 448.0, -779.0, 1.6623022556304932]], 'tvmonitor': [[0.0, 0.0, 448.0, 85.0, 3.238999128341675], [240.0, 772.0, 200.0, 91.0, 1.7443398237228394], [546.0, 155.0, 448.0, 448.0, 1.1334525346755981], [107.0, 441.0, 432.0, 219.0, 0.5971617698669434]], 'chair': [[470.0, -187.0, 106.0, 235.0, 3.8548083305358887], [524.0, 740.0, -103.0, 99.0, 3.636549234390259], [0.0, 0.0, 275.0, -325.0, 3.0997846126556396], [711.0, -231.0, -146.0, 392.0, 2.205275535583496]], 'diningtable': [[138.0, -310.0, 111.0, 448.0, 4.660728931427002], [317.0, -66.0, 313.0, 6.0, 4.535496234893799], [0.0, 0.0, -41.0, 175.0, 1.8571208715438843], [21.0, -92.0, 76.0, 172.0, 1.2035608291625977], [0.0, 0.0, 448.0, -250.0, 1.00322687625885]], 'car': [[312.0, 232.0, 132.0, 309.0, 3.205225706100464], [514.0, -76.0, 218.0, 448.0, 1.4289973974227905], [0.0, 0.0, 448.0, 142.0, 0.7124998569488525]]} WHile I expect only the dict to contain the small_ball key How's that is possible? where's the prediction output?How to fix the code? From list1 at tompassin.net Tue Jul 30 15:25:39 2024 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 30 Jul 2024 15:25:39 -0400 Subject: Predicting an object over an pretrained model is not working In-Reply-To: References: Message-ID: <263356ef-7ad8-4abc-9940-bd8536ee13eb@tompassin.net> On 7/30/2024 2:18 PM, marc nicole via Python-list wrote: > Hello all, > > I want to predict an object by given as input an image and want to have my > model be able to predict the label. I have trained a model using tensorflow > based on annotated database where the target object to predict was added to > the pretrained model. the code I am using is the following where I set the > target object image as input and want to have the prediction output: > > > > > > > > > class MultiObjectDetection(): > > def __init__(self, classes_name): > > self._classes_name = classes_name > self._num_classes = len(classes_name) > > self._common_params = {'image_size': 448, 'num_classes': > self._num_classes, > 'batch_size':1} > self._net_params = {'cell_size': 7, 'boxes_per_cell':2, > 'weight_decay': 0.0005} > self._net = YoloTinyNet(self._common_params, self._net_params, > test=True) > > def predict_object(self, image): > predicts = self._net.inference(image) > return predicts > > def process_predicts(self, resized_img, predicts, thresh=0.2): > """ > process the predicts of object detection with one image input. > > Args: > resized_img: resized source image. > predicts: output of the model. > thresh: thresh of bounding box confidence. > Return: > predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], [...]]}. > """ > cls_num = self._num_classes > bbx_per_cell = self._net_params["boxes_per_cell"] > cell_size = self._net_params["cell_size"] > img_size = self._common_params["image_size"] > p_classes = predicts[0, :, :, 0:cls_num] > C = predicts[0, :, :, cls_num:cls_num+bbx_per_cell] # two > bounding boxes in one cell. > coordinate = predicts[0, :, :, cls_num+bbx_per_cell:] # all > bounding boxes position. > > p_classes = np.reshape(p_classes, (cell_size, cell_size, 1, cls_num)) > C = np.reshape(C, (cell_size, cell_size, bbx_per_cell, 1)) > > P = C * p_classes # confidencefor all classes of all bounding > boxes (cell_size, cell_size, bounding_box_num, class_num) = (7, 7, 2, > 1). > > predicts_dict = {} > for i in range(cell_size): > for j in range(cell_size): > temp_data = np.zeros_like(P, np.float32) > temp_data[i, j, :, :] = P[i, j, :, :] > position = np.argmax(temp_data) # refer to the class > num (with maximum confidence) for every bounding box. > index = np.unravel_index(position, P.shape) > > if P[index] > thresh: > class_num = index[-1] > coordinate = np.reshape(coordinate, (cell_size, > cell_size, bbx_per_cell, 4)) # (cell_size, cell_size, > bbox_num_per_cell, coordinate)[xmin, ymin, xmax, ymax] > max_coordinate = coordinate[index[0], index[1], index[2], :] > > xcenter = max_coordinate[0] > ycenter = max_coordinate[1] > w = max_coordinate[2] > h = max_coordinate[3] > > xcenter = (index[1] + xcenter) * (1.0*img_size /cell_size) > ycenter = (index[0] + ycenter) * (1.0*img_size /cell_size) > > w = w * img_size > h = h * img_size > xmin = 0 if (xcenter - w/2.0 < 0) else (xcenter - w/2.0) > ymin = 0 if (xcenter - w/2.0 < 0) else (ycenter - h/2.0) > xmax = resized_img.shape[0] if (xmin + w) > > resized_img.shape[0] else (xmin + w) > ymax = resized_img.shape[1] if (ymin + h) > > resized_img.shape[1] else (ymin + h) > > class_name = self._classes_name[class_num] > predicts_dict.setdefault(class_name, []) > predicts_dict[class_name].append([int(xmin), > int(ymin), int(xmax), int(ymax), P[index]]) > > return predicts_dict > > def non_max_suppress(self, predicts_dict, threshold=0.5): > """ > implement non-maximum supression on predict bounding boxes. > Args: > predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], [...]]}. > threshhold: iou threshold > Return: > predicts_dict processed by non-maximum suppression > """ > for object_name, bbox in predicts_dict.items(): > bbox_array = np.array(bbox, dtype=np.float) > x1, y1, x2, y2, scores = bbox_array[:,0], bbox_array[:,1], > bbox_array[:,2], bbox_array[:,3], bbox_array[:,4] > areas = (x2-x1+1) * (y2-y1+1) > order = scores.argsort()[::-1] > keep = [] > while order.size > 0: > i = order[0] > keep.append(i) > xx1 = np.maximum(x1[i], x1[order[1:]]) > yy1 = np.maximum(y1[i], y1[order[1:]]) > xx2 = np.minimum(x2[i], x2[order[1:]]) > yy2 = np.minimum(y2[i], y2[order[1:]]) > inter = np.maximum(0.0, xx2-xx1+1) * np.maximum(0.0, yy2-yy1+1) > iou = inter/(areas[i]+areas[order[1:]]-inter) > indexs = np.where(iou<=threshold)[0] > order = order[indexs+1] > bbox = bbox_array[keep] > predicts_dict[object_name] = bbox.tolist() > predicts_dict = predicts_dict > return predicts_dict > > > > class_names = ["aeroplane", "bicycle", "bird", "boat", "bottle", > "bus", "car", "cat", "chair", "cow", "diningtable", > "dog", "horse", "motorbike", "person", > "pottedplant", "sheep", "sofa", "train", "tvmonitor", > "small_ball"] > modelFile = ('models\\train\\model.ckpt-0') > track_object = "small_ball"print("object detection and tracking...") > > multiObjectDetect = MultiObjectDetection(IP, class_names) > image = tf.placeholder(tf.float32, (1, 448, 448, 3)) > object_predicts = multiObjectDetect.predict_object(image) > > > > sess = tf.Session() > saver = tf.train.Saver(multiObjectDetect._net.trainable_collection) > > > saver.restore(sess, modelFile) > > index = 0while 1: > > src_img = cv2.imread("./weirdobject.jpg") > resized_img = cv2.resize(src_img, (448, 448)) > > np_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2RGB) > np_img = np_img.astype(np.float32) > np_img = np_img / 255.0 * 2 - 1 > np_img = np.reshape(np_img, (1, 448, 448, 3)) > > > np_predict = sess.run(object_predicts, feed_dict={image: np_img}) > predicts_dict = multiObjectDetect.process_predicts(resized_img, np_predict) > predicts_dict = multiObjectDetect.non_max_suppress(predicts_dict) > > print ("predict dict = ", predicts_dict) > > > > > > > > The problem with this code is that the predicts_dict returns: > > > > predict dict = {'sheep': [[233.0, 92.0, 448.0, -103.0, > 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], > [0.0, 0.0, 448.0, 431.0, 3.393721580505371]], 'horse': [[374.0, 33.0, > 282.0, 448.0, 5.277851581573486], [135.0, 688.0, -33.0, -14.0, > 3.5144259929656982], [1.0, 117.0, 112.0, -138.0, 2.656987190246582]], > 'bicycle': [[461.0, 781.0, 154.0, -381.0, 5.918102741241455], [70.0, > 344.0, 391.0, -138.0, 3.031444787979126], [378.0, 497.0, 46.0, 149.0, > 2.7629122734069824], [541.0, 583.0, 69.0, 307.0, 2.7170517444610596], > [323.0, 22.0, 336.0, 448.0, 1.608760952949524]], 'bottle': [[390.0, > 218.0, -199.0, 448.0, 4.582971096038818], [0.0, 0.0, 448.0, -410.0, > 0.9097045063972473]], 'sofa': [[346.0, 102.0, 323.0, -38.0, > 2.371835947036743]], 'dog': [[319.0, 254.0, -282.0, 373.0, > 4.022889137268066]], 'cat': [[63.0, -195.0, 365.0, -92.0, > 3.5134828090667725]], 'person': [[22.0, -122.0, 154.0, 448.0, > 3.927537441253662], [350.0, 155.0, -36.0, -445.0, 2.679833173751831], > [119.0, 416.0, -43.0, 292.0, 0.9529445171356201], [251.0, 445.0, > 225.0, 188.0, 0.9001350402832031]], 'train': [[329.0, 485.0, -24.0, > -235.0, 2.7050414085388184], [483.0, 362.0, 237.0, -86.0, > 2.555817127227783], [13.0, 365.0, 373.0, 448.0, 0.6229299902915955]], > 'small_ball': [[217.0, 737.0, 448.0, -315.0, 1.739920973777771], > [117.0, 283.0, 153.0, 122.0, 1.5690066814422607]], 'boat': [[164.0, > 805.0, 34.0, -169.0, 4.972668170928955], [0.0, 0.0, 397.0, 69.0, > 2.353729486465454], [302.0, 605.0, 15.0, -22.0, 2.0259625911712646]], > 'aeroplane': [[470.0, 616.0, -305.0, -37.0, 3.431873321533203], [0.0, > 0.0, 448.0, -72.0, 2.836672306060791]], 'bus': [[0.0, 0.0, -101.0, > -280.0, 1.2078320980072021]], 'pottedplant': [[620.0, -268.0, -124.0, > 418.0, 2.158564805984497], [0.0, 0.0, 448.0, -779.0, > 1.6623022556304932]], 'tvmonitor': [[0.0, 0.0, 448.0, 85.0, > 3.238999128341675], [240.0, 772.0, 200.0, 91.0, 1.7443398237228394], > [546.0, 155.0, 448.0, 448.0, 1.1334525346755981], [107.0, 441.0, > 432.0, 219.0, 0.5971617698669434]], 'chair': [[470.0, -187.0, 106.0, > 235.0, 3.8548083305358887], [524.0, 740.0, -103.0, 99.0, > 3.636549234390259], [0.0, 0.0, 275.0, -325.0, 3.0997846126556396], > [711.0, -231.0, -146.0, 392.0, 2.205275535583496]], 'diningtable': > [[138.0, -310.0, 111.0, 448.0, 4.660728931427002], [317.0, -66.0, > 313.0, 6.0, 4.535496234893799], [0.0, 0.0, -41.0, 175.0, > 1.8571208715438843], [21.0, -92.0, 76.0, 172.0, 1.2035608291625977], > [0.0, 0.0, 448.0, -250.0, 1.00322687625885]], 'car': [[312.0, 232.0, > 132.0, 309.0, 3.205225706100464], [514.0, -76.0, 218.0, 448.0, > 1.4289973974227905], [0.0, 0.0, 448.0, 142.0, 0.7124998569488525]]} > > > WHile I expect only the dict to contain the small_ball key > > > > How's that is possible? where's the prediction output?How to fix the code? Without trying to figure out all that code, why would you expect only results for a single key? An ML system is going to compute probabilities and parameters for all objects it knows about (presumably subject to some threshold). From mk1853387 at gmail.com Tue Jul 30 16:49:21 2024 From: mk1853387 at gmail.com (marc nicole) Date: Tue, 30 Jul 2024 22:49:21 +0200 Subject: Predicting an object over an pretrained model is not working In-Reply-To: <263356ef-7ad8-4abc-9940-bd8536ee13eb@tompassin.net> References: <263356ef-7ad8-4abc-9940-bd8536ee13eb@tompassin.net> Message-ID: OK, but how's the probability of small_ball greater than others? I can't find it anyway, what's its value? Le mar. 30 juil. 2024 ? 21:37, Thomas Passin via Python-list < python-list at python.org> a ?crit : > On 7/30/2024 2:18 PM, marc nicole via Python-list wrote: > > Hello all, > > > > I want to predict an object by given as input an image and want to have > my > > model be able to predict the label. I have trained a model using > tensorflow > > based on annotated database where the target object to predict was added > to > > the pretrained model. the code I am using is the following where I set > the > > target object image as input and want to have the prediction output: > > > > > > > > > > > > > > > > > > class MultiObjectDetection(): > > > > def __init__(self, classes_name): > > > > self._classes_name = classes_name > > self._num_classes = len(classes_name) > > > > self._common_params = {'image_size': 448, 'num_classes': > > self._num_classes, > > 'batch_size':1} > > self._net_params = {'cell_size': 7, 'boxes_per_cell':2, > > 'weight_decay': 0.0005} > > self._net = YoloTinyNet(self._common_params, self._net_params, > > test=True) > > > > def predict_object(self, image): > > predicts = self._net.inference(image) > > return predicts > > > > def process_predicts(self, resized_img, predicts, thresh=0.2): > > """ > > process the predicts of object detection with one image input. > > > > Args: > > resized_img: resized source image. > > predicts: output of the model. > > thresh: thresh of bounding box confidence. > > Return: > > predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], > [...]]}. > > """ > > cls_num = self._num_classes > > bbx_per_cell = self._net_params["boxes_per_cell"] > > cell_size = self._net_params["cell_size"] > > img_size = self._common_params["image_size"] > > p_classes = predicts[0, :, :, 0:cls_num] > > C = predicts[0, :, :, cls_num:cls_num+bbx_per_cell] # two > > bounding boxes in one cell. > > coordinate = predicts[0, :, :, cls_num+bbx_per_cell:] # all > > bounding boxes position. > > > > p_classes = np.reshape(p_classes, (cell_size, cell_size, 1, > cls_num)) > > C = np.reshape(C, (cell_size, cell_size, bbx_per_cell, 1)) > > > > P = C * p_classes # confidencefor all classes of all bounding > > boxes (cell_size, cell_size, bounding_box_num, class_num) = (7, 7, 2, > > 1). > > > > predicts_dict = {} > > for i in range(cell_size): > > for j in range(cell_size): > > temp_data = np.zeros_like(P, np.float32) > > temp_data[i, j, :, :] = P[i, j, :, :] > > position = np.argmax(temp_data) # refer to the class > > num (with maximum confidence) for every bounding box. > > index = np.unravel_index(position, P.shape) > > > > if P[index] > thresh: > > class_num = index[-1] > > coordinate = np.reshape(coordinate, (cell_size, > > cell_size, bbx_per_cell, 4)) # (cell_size, cell_size, > > bbox_num_per_cell, coordinate)[xmin, ymin, xmax, ymax] > > max_coordinate = coordinate[index[0], index[1], > index[2], :] > > > > xcenter = max_coordinate[0] > > ycenter = max_coordinate[1] > > w = max_coordinate[2] > > h = max_coordinate[3] > > > > xcenter = (index[1] + xcenter) * (1.0*img_size > /cell_size) > > ycenter = (index[0] + ycenter) * (1.0*img_size > /cell_size) > > > > w = w * img_size > > h = h * img_size > > xmin = 0 if (xcenter - w/2.0 < 0) else (xcenter - > w/2.0) > > ymin = 0 if (xcenter - w/2.0 < 0) else (ycenter - > h/2.0) > > xmax = resized_img.shape[0] if (xmin + w) > > > resized_img.shape[0] else (xmin + w) > > ymax = resized_img.shape[1] if (ymin + h) > > > resized_img.shape[1] else (ymin + h) > > > > class_name = self._classes_name[class_num] > > predicts_dict.setdefault(class_name, []) > > predicts_dict[class_name].append([int(xmin), > > int(ymin), int(xmax), int(ymax), P[index]]) > > > > return predicts_dict > > > > def non_max_suppress(self, predicts_dict, threshold=0.5): > > """ > > implement non-maximum supression on predict bounding boxes. > > Args: > > predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], > [...]]}. > > threshhold: iou threshold > > Return: > > predicts_dict processed by non-maximum suppression > > """ > > for object_name, bbox in predicts_dict.items(): > > bbox_array = np.array(bbox, dtype=np.float) > > x1, y1, x2, y2, scores = bbox_array[:,0], bbox_array[:,1], > > bbox_array[:,2], bbox_array[:,3], bbox_array[:,4] > > areas = (x2-x1+1) * (y2-y1+1) > > order = scores.argsort()[::-1] > > keep = [] > > while order.size > 0: > > i = order[0] > > keep.append(i) > > xx1 = np.maximum(x1[i], x1[order[1:]]) > > yy1 = np.maximum(y1[i], y1[order[1:]]) > > xx2 = np.minimum(x2[i], x2[order[1:]]) > > yy2 = np.minimum(y2[i], y2[order[1:]]) > > inter = np.maximum(0.0, xx2-xx1+1) * np.maximum(0.0, > yy2-yy1+1) > > iou = inter/(areas[i]+areas[order[1:]]-inter) > > indexs = np.where(iou<=threshold)[0] > > order = order[indexs+1] > > bbox = bbox_array[keep] > > predicts_dict[object_name] = bbox.tolist() > > predicts_dict = predicts_dict > > return predicts_dict > > > > > > > > class_names = ["aeroplane", "bicycle", "bird", "boat", "bottle", > > "bus", "car", "cat", "chair", "cow", "diningtable", > > "dog", "horse", "motorbike", "person", > > "pottedplant", "sheep", "sofa", "train", "tvmonitor", > > "small_ball"] > > modelFile = ('models\\train\\model.ckpt-0') > > track_object = "small_ball"print("object detection and tracking...") > > > > multiObjectDetect = MultiObjectDetection(IP, class_names) > > image = tf.placeholder(tf.float32, (1, 448, 448, 3)) > > object_predicts = multiObjectDetect.predict_object(image) > > > > > > > > sess = tf.Session() > > saver = tf.train.Saver(multiObjectDetect._net.trainable_collection) > > > > > > saver.restore(sess, modelFile) > > > > index = 0while 1: > > > > src_img = cv2.imread("./weirdobject.jpg") > > resized_img = cv2.resize(src_img, (448, 448)) > > > > np_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2RGB) > > np_img = np_img.astype(np.float32) > > np_img = np_img / 255.0 * 2 - 1 > > np_img = np.reshape(np_img, (1, 448, 448, 3)) > > > > > > np_predict = sess.run(object_predicts, feed_dict={image: np_img}) > > predicts_dict = multiObjectDetect.process_predicts(resized_img, > np_predict) > > predicts_dict = multiObjectDetect.non_max_suppress(predicts_dict) > > > > print ("predict dict = ", predicts_dict) > > > > > > > > > > > > > > > > The problem with this code is that the predicts_dict returns: > > > > > > > > predict dict = {'sheep': [[233.0, 92.0, 448.0, -103.0, > > 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], > > [0.0, 0.0, 448.0, 431.0, 3.393721580505371]], 'horse': [[374.0, 33.0, > > 282.0, 448.0, 5.277851581573486], [135.0, 688.0, -33.0, -14.0, > > 3.5144259929656982], [1.0, 117.0, 112.0, -138.0, 2.656987190246582]], > > 'bicycle': [[461.0, 781.0, 154.0, -381.0, 5.918102741241455], [70.0, > > 344.0, 391.0, -138.0, 3.031444787979126], [378.0, 497.0, 46.0, 149.0, > > 2.7629122734069824], [541.0, 583.0, 69.0, 307.0, 2.7170517444610596], > > [323.0, 22.0, 336.0, 448.0, 1.608760952949524]], 'bottle': [[390.0, > > 218.0, -199.0, 448.0, 4.582971096038818], [0.0, 0.0, 448.0, -410.0, > > 0.9097045063972473]], 'sofa': [[346.0, 102.0, 323.0, -38.0, > > 2.371835947036743]], 'dog': [[319.0, 254.0, -282.0, 373.0, > > 4.022889137268066]], 'cat': [[63.0, -195.0, 365.0, -92.0, > > 3.5134828090667725]], 'person': [[22.0, -122.0, 154.0, 448.0, > > 3.927537441253662], [350.0, 155.0, -36.0, -445.0, 2.679833173751831], > > [119.0, 416.0, -43.0, 292.0, 0.9529445171356201], [251.0, 445.0, > > 225.0, 188.0, 0.9001350402832031]], 'train': [[329.0, 485.0, -24.0, > > -235.0, 2.7050414085388184], [483.0, 362.0, 237.0, -86.0, > > 2.555817127227783], [13.0, 365.0, 373.0, 448.0, 0.6229299902915955]], > > 'small_ball': [[217.0, 737.0, 448.0, -315.0, 1.739920973777771], > > [117.0, 283.0, 153.0, 122.0, 1.5690066814422607]], 'boat': [[164.0, > > 805.0, 34.0, -169.0, 4.972668170928955], [0.0, 0.0, 397.0, 69.0, > > 2.353729486465454], [302.0, 605.0, 15.0, -22.0, 2.0259625911712646]], > > 'aeroplane': [[470.0, 616.0, -305.0, -37.0, 3.431873321533203], [0.0, > > 0.0, 448.0, -72.0, 2.836672306060791]], 'bus': [[0.0, 0.0, -101.0, > > -280.0, 1.2078320980072021]], 'pottedplant': [[620.0, -268.0, -124.0, > > 418.0, 2.158564805984497], [0.0, 0.0, 448.0, -779.0, > > 1.6623022556304932]], 'tvmonitor': [[0.0, 0.0, 448.0, 85.0, > > 3.238999128341675], [240.0, 772.0, 200.0, 91.0, 1.7443398237228394], > > [546.0, 155.0, 448.0, 448.0, 1.1334525346755981], [107.0, 441.0, > > 432.0, 219.0, 0.5971617698669434]], 'chair': [[470.0, -187.0, 106.0, > > 235.0, 3.8548083305358887], [524.0, 740.0, -103.0, 99.0, > > 3.636549234390259], [0.0, 0.0, 275.0, -325.0, 3.0997846126556396], > > [711.0, -231.0, -146.0, 392.0, 2.205275535583496]], 'diningtable': > > [[138.0, -310.0, 111.0, 448.0, 4.660728931427002], [317.0, -66.0, > > 313.0, 6.0, 4.535496234893799], [0.0, 0.0, -41.0, 175.0, > > 1.8571208715438843], [21.0, -92.0, 76.0, 172.0, 1.2035608291625977], > > [0.0, 0.0, 448.0, -250.0, 1.00322687625885]], 'car': [[312.0, 232.0, > > 132.0, 309.0, 3.205225706100464], [514.0, -76.0, 218.0, 448.0, > > 1.4289973974227905], [0.0, 0.0, 448.0, 142.0, 0.7124998569488525]]} > > > > > > WHile I expect only the dict to contain the small_ball key > > > > > > > > How's that is possible? where's the prediction output?How to fix the > code? > > Without trying to figure out all that code, why would you expect only > results for a single key? An ML system is going to compute > probabilities and parameters for all objects it knows about (presumably > subject to some threshold). > > -- > https://mail.python.org/mailman/listinfo/python-list > From list1 at tompassin.net Tue Jul 30 17:45:20 2024 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 30 Jul 2024 17:45:20 -0400 Subject: Predicting an object over an pretrained model is not working In-Reply-To: References: <263356ef-7ad8-4abc-9940-bd8536ee13eb@tompassin.net> Message-ID: On 7/30/2024 4:49 PM, marc nicole wrote: > OK, but how's the probability of small_ball greater than others? I can't > find it anyway, what's its value? It's your code. I wouldn't know. I suppose it's represented somewhere in all those parameters. You need to understand what those function calls are returning. It's documented somewhere, right? And you really do need to know the probabilities of the competing images because otherwise you won't know how confident you can be that the identification is a strong one. > Le?mar. 30 juil. 2024 ??21:37, Thomas Passin via Python-list > > a ?crit?: > > On 7/30/2024 2:18 PM, marc nicole via Python-list wrote: > > Hello all, > > > > I want to predict an object by given as input an image and want > to have my > > model be able to predict the label. I have trained a model using > tensorflow > > based on annotated database where the target object to predict > was added to > > the pretrained model. the code I am using is the following where > I set the > > target object image as input and want to have the prediction output: > > > > > > > > > > > > > > > > > > class MultiObjectDetection(): > > > >? ? ? def __init__(self, classes_name): > > > >? ? ? ? ? self._classes_name = classes_name > >? ? ? ? ? self._num_classes = len(classes_name) > > > >? ? ? ? ? self._common_params = {'image_size': 448, 'num_classes': > > self._num_classes, > >? ? ? ? ? ? ? ? ? 'batch_size':1} > >? ? ? ? ? self._net_params = {'cell_size': 7, 'boxes_per_cell':2, > > 'weight_decay': 0.0005} > >? ? ? ? ? self._net = YoloTinyNet(self._common_params, > self._net_params, > > test=True) > > > >? ? ? def predict_object(self, image): > >? ? ? ? ? predicts = self._net.inference(image) > >? ? ? ? ? return predicts > > > >? ? ? def process_predicts(self, resized_img, predicts, thresh=0.2): > >? ? ? ? ? """ > >? ? ? ? ? process the predicts of object detection with one image > input. > > > >? ? ? ? ? Args: > >? ? ? ? ? ? ? resized_img: resized source image. > >? ? ? ? ? ? ? predicts: output of the model. > >? ? ? ? ? ? ? thresh: thresh of bounding box confidence. > >? ? ? ? ? Return: > >? ? ? ? ? ? ? predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], > [...]]}. > >? ? ? ? ? """ > >? ? ? ? ? cls_num = self._num_classes > >? ? ? ? ? bbx_per_cell = self._net_params["boxes_per_cell"] > >? ? ? ? ? cell_size = self._net_params["cell_size"] > >? ? ? ? ? img_size = self._common_params["image_size"] > >? ? ? ? ? p_classes = predicts[0, :, :, 0:cls_num] > >? ? ? ? ? C = predicts[0, :, :, cls_num:cls_num+bbx_per_cell] # two > > bounding boxes in one cell. > >? ? ? ? ? coordinate = predicts[0, :, :, cls_num+bbx_per_cell:] # all > > bounding boxes position. > > > >? ? ? ? ? p_classes = np.reshape(p_classes, (cell_size, cell_size, > 1, cls_num)) > >? ? ? ? ? C = np.reshape(C, (cell_size, cell_size, bbx_per_cell, 1)) > > > >? ? ? ? ? P = C * p_classes # confidencefor all classes of all > bounding > > boxes (cell_size, cell_size, bounding_box_num, class_num) = (7, 7, 2, > > 1). > > > >? ? ? ? ? predicts_dict = {} > >? ? ? ? ? for i in range(cell_size): > >? ? ? ? ? ? ? for j in range(cell_size): > >? ? ? ? ? ? ? ? ? temp_data = np.zeros_like(P, np.float32) > >? ? ? ? ? ? ? ? ? temp_data[i, j, :, :] = P[i, j, :, :] > >? ? ? ? ? ? ? ? ? position = np.argmax(temp_data) # refer to the class > > num (with maximum confidence) for every bounding box. > >? ? ? ? ? ? ? ? ? index = np.unravel_index(position, P.shape) > > > >? ? ? ? ? ? ? ? ? if P[index] > thresh: > >? ? ? ? ? ? ? ? ? ? ? class_num = index[-1] > >? ? ? ? ? ? ? ? ? ? ? coordinate = np.reshape(coordinate, (cell_size, > > cell_size, bbx_per_cell, 4)) # (cell_size, cell_size, > > bbox_num_per_cell, coordinate)[xmin, ymin, xmax, ymax] > >? ? ? ? ? ? ? ? ? ? ? max_coordinate = coordinate[index[0], > index[1], index[2], :] > > > >? ? ? ? ? ? ? ? ? ? ? xcenter = max_coordinate[0] > >? ? ? ? ? ? ? ? ? ? ? ycenter = max_coordinate[1] > >? ? ? ? ? ? ? ? ? ? ? w = max_coordinate[2] > >? ? ? ? ? ? ? ? ? ? ? h = max_coordinate[3] > > > >? ? ? ? ? ? ? ? ? ? ? xcenter = (index[1] + xcenter) * > (1.0*img_size /cell_size) > >? ? ? ? ? ? ? ? ? ? ? ycenter = (index[0] + ycenter) * > (1.0*img_size /cell_size) > > > >? ? ? ? ? ? ? ? ? ? ? w = w * img_size > >? ? ? ? ? ? ? ? ? ? ? h = h * img_size > >? ? ? ? ? ? ? ? ? ? ? xmin = 0 if (xcenter - w/2.0 < 0) else > (xcenter - w/2.0) > >? ? ? ? ? ? ? ? ? ? ? ymin = 0 if (xcenter - w/2.0 < 0) else > (ycenter - h/2.0) > >? ? ? ? ? ? ? ? ? ? ? xmax = resized_img.shape[0] if (xmin + w) > > > resized_img.shape[0] else (xmin + w) > >? ? ? ? ? ? ? ? ? ? ? ymax = resized_img.shape[1] if (ymin + h) > > > resized_img.shape[1] else (ymin + h) > > > >? ? ? ? ? ? ? ? ? ? ? class_name = self._classes_name[class_num] > >? ? ? ? ? ? ? ? ? ? ? predicts_dict.setdefault(class_name, []) > >? ? ? ? ? ? ? ? ? ? ? predicts_dict[class_name].append([int(xmin), > > int(ymin), int(xmax), int(ymax), P[index]]) > > > >? ? ? ? ? return predicts_dict > > > >? ? ? def non_max_suppress(self, predicts_dict, threshold=0.5): > >? ? ? ? ? """ > >? ? ? ? ? implement non-maximum supression on predict bounding boxes. > >? ? ? ? ? Args: > >? ? ? ? ? ? ? predicts_dict: {"stick": [[x1, y1, x2, y2, scores1], > [...]]}. > >? ? ? ? ? ? ? threshhold: iou threshold > >? ? ? ? ? Return: > >? ? ? ? ? ? ? predicts_dict processed by non-maximum suppression > >? ? ? ? ? """ > >? ? ? ? ? for object_name, bbox in predicts_dict.items(): > >? ? ? ? ? ? ? bbox_array = np.array(bbox, dtype=np.float) > >? ? ? ? ? ? ? x1, y1, x2, y2, scores = bbox_array[:,0], > bbox_array[:,1], > > bbox_array[:,2], bbox_array[:,3], bbox_array[:,4] > >? ? ? ? ? ? ? areas = (x2-x1+1) * (y2-y1+1) > >? ? ? ? ? ? ? order = scores.argsort()[::-1] > >? ? ? ? ? ? ? keep = [] > >? ? ? ? ? ? ? while order.size > 0: > >? ? ? ? ? ? ? ? ? i = order[0] > >? ? ? ? ? ? ? ? ? keep.append(i) > >? ? ? ? ? ? ? ? ? xx1 = np.maximum(x1[i], x1[order[1:]]) > >? ? ? ? ? ? ? ? ? yy1 = np.maximum(y1[i], y1[order[1:]]) > >? ? ? ? ? ? ? ? ? xx2 = np.minimum(x2[i], x2[order[1:]]) > >? ? ? ? ? ? ? ? ? yy2 = np.minimum(y2[i], y2[order[1:]]) > >? ? ? ? ? ? ? ? ? inter = np.maximum(0.0, xx2-xx1+1) * > np.maximum(0.0, yy2-yy1+1) > >? ? ? ? ? ? ? ? ? iou = inter/(areas[i]+areas[order[1:]]-inter) > >? ? ? ? ? ? ? ? ? indexs = np.where(iou<=threshold)[0] > >? ? ? ? ? ? ? ? ? order = order[indexs+1] > >? ? ? ? ? ? ? bbox = bbox_array[keep] > >? ? ? ? ? ? ? predicts_dict[object_name] = bbox.tolist() > >? ? ? ? ? ? ? predicts_dict = predicts_dict > >? ? ? ? ? return predicts_dict > > > > > > > > class_names = ["aeroplane", "bicycle", "bird", "boat", "bottle", > > "bus", "car", "cat", "chair", "cow", "diningtable", > >? ? ? ? ? ? ? ? ? ? ?"dog", "horse", "motorbike", "person", > > "pottedplant", "sheep", "sofa", "train", "tvmonitor", > >? ? ? ? ? ? ? ? ? ? ?"small_ball"] > > modelFile = ('models\\train\\model.ckpt-0') > > track_object = "small_ball"print("object detection and tracking...") > > > > multiObjectDetect = MultiObjectDetection(IP, class_names) > > image = tf.placeholder(tf.float32, (1, 448, 448, 3)) > > object_predicts = multiObjectDetect.predict_object(image) > > > > > > > > sess = tf.Session() > > saver = tf.train.Saver(multiObjectDetect._net.trainable_collection) > > > > > > saver.restore(sess, modelFile) > > > > index = 0while 1: > > > >? ? ? src_img = cv2.imread("./weirdobject.jpg") > >? ? ? resized_img = cv2.resize(src_img, (448, 448)) > > > >? ? ? np_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2RGB) > >? ? ? np_img = np_img.astype(np.float32) > >? ? ? np_img = np_img / 255.0 * 2 - 1 > >? ? ? np_img = np.reshape(np_img, (1, 448, 448, 3)) > > > > > >? ? ? np_predict = sess.run(object_predicts, feed_dict={image: > np_img}) > >? ? ? predicts_dict = > multiObjectDetect.process_predicts(resized_img, np_predict) > >? ? ? predicts_dict = > multiObjectDetect.non_max_suppress(predicts_dict) > > > >? ? ? print ("predict dict = ", predicts_dict) > > > > > > > > > > > > > > > > The problem with this code is that the predicts_dict returns: > > > > > > > > predict dict =? {'sheep': [[233.0, 92.0, 448.0, -103.0, > > 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], > > [0.0, 0.0, 448.0, 431.0, 3.393721580505371]], 'horse': [[374.0, 33.0, > > 282.0, 448.0, 5.277851581573486], [135.0, 688.0, -33.0, -14.0, > > 3.5144259929656982], [1.0, 117.0, 112.0, -138.0, 2.656987190246582]], > > 'bicycle': [[461.0, 781.0, 154.0, -381.0, 5.918102741241455], [70.0, > > 344.0, 391.0, -138.0, 3.031444787979126], [378.0, 497.0, 46.0, 149.0, > > 2.7629122734069824], [541.0, 583.0, 69.0, 307.0, 2.7170517444610596], > > [323.0, 22.0, 336.0, 448.0, 1.608760952949524]], 'bottle': [[390.0, > > 218.0, -199.0, 448.0, 4.582971096038818], [0.0, 0.0, 448.0, -410.0, > > 0.9097045063972473]], 'sofa': [[346.0, 102.0, 323.0, -38.0, > > 2.371835947036743]], 'dog': [[319.0, 254.0, -282.0, 373.0, > > 4.022889137268066]], 'cat': [[63.0, -195.0, 365.0, -92.0, > > 3.5134828090667725]], 'person': [[22.0, -122.0, 154.0, 448.0, > > 3.927537441253662], [350.0, 155.0, -36.0, -445.0, 2.679833173751831], > > [119.0, 416.0, -43.0, 292.0, 0.9529445171356201], [251.0, 445.0, > > 225.0, 188.0, 0.9001350402832031]], 'train': [[329.0, 485.0, -24.0, > > -235.0, 2.7050414085388184], [483.0, 362.0, 237.0, -86.0, > > 2.555817127227783], [13.0, 365.0, 373.0, 448.0, 0.6229299902915955]], > > 'small_ball': [[217.0, 737.0, 448.0, -315.0, 1.739920973777771], > > [117.0, 283.0, 153.0, 122.0, 1.5690066814422607]], 'boat': [[164.0, > > 805.0, 34.0, -169.0, 4.972668170928955], [0.0, 0.0, 397.0, 69.0, > > 2.353729486465454], [302.0, 605.0, 15.0, -22.0, 2.0259625911712646]], > > 'aeroplane': [[470.0, 616.0, -305.0, -37.0, 3.431873321533203], [0.0, > > 0.0, 448.0, -72.0, 2.836672306060791]], 'bus': [[0.0, 0.0, -101.0, > > -280.0, 1.2078320980072021]], 'pottedplant': [[620.0, -268.0, -124.0, > > 418.0, 2.158564805984497], [0.0, 0.0, 448.0, -779.0, > > 1.6623022556304932]], 'tvmonitor': [[0.0, 0.0, 448.0, 85.0, > > 3.238999128341675], [240.0, 772.0, 200.0, 91.0, 1.7443398237228394], > > [546.0, 155.0, 448.0, 448.0, 1.1334525346755981], [107.0, 441.0, > > 432.0, 219.0, 0.5971617698669434]], 'chair': [[470.0, -187.0, 106.0, > > 235.0, 3.8548083305358887], [524.0, 740.0, -103.0, 99.0, > > 3.636549234390259], [0.0, 0.0, 275.0, -325.0, 3.0997846126556396], > > [711.0, -231.0, -146.0, 392.0, 2.205275535583496]], 'diningtable': > > [[138.0, -310.0, 111.0, 448.0, 4.660728931427002], [317.0, -66.0, > > 313.0, 6.0, 4.535496234893799], [0.0, 0.0, -41.0, 175.0, > > 1.8571208715438843], [21.0, -92.0, 76.0, 172.0, 1.2035608291625977], > > [0.0, 0.0, 448.0, -250.0, 1.00322687625885]], 'car': [[312.0, 232.0, > > 132.0, 309.0, 3.205225706100464], [514.0, -76.0, 218.0, 448.0, > > 1.4289973974227905], [0.0, 0.0, 448.0, 142.0, 0.7124998569488525]]} > > > > > > WHile I expect only the dict to contain the small_ball key > > > > > > > > How's that is possible? where's the prediction output?How to fix > the code? > > Without trying to figure out all that code, why would you expect only > results for a single key?? An ML system is going to compute > probabilities and parameters for all objects it knows about (presumably > subject to some threshold). > > -- > https://mail.python.org/mailman/listinfo/python-list > > From PythonList at DancesWithMice.info Tue Jul 30 18:16:29 2024 From: PythonList at DancesWithMice.info (dn) Date: Wed, 31 Jul 2024 10:16:29 +1200 Subject: Predicting an object over an pretrained model is not working In-Reply-To: References: Message-ID: <339c86d5-8cb5-4995-b5ba-12f88d71a107@DancesWithMice.info> On 31/07/24 06:18, marc nicole via Python-list wrote: > Hello all, > > I want to predict an object by given as input an image and want to have my > model be able to predict the label. I have trained a model using tensorflow > based on annotated database where the target object to predict was added to > the pretrained model. the code I am using is the following where I set the > target object image as input and want to have the prediction output: ... > WHile I expect only the dict to contain the small_ball key > How's that is possible? where's the prediction output?How to fix the code? To save us lots of reading and study to be able to help you, please advise: 1 what are the meanings of all these numbers? > 'sheep': [[233.0, 92.0, 448.0, -103.0, >> 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], >> [0.0, 0.0, 448.0, 431.0, 3.393721580505371]] 2 (assuming it hasn't) why the dict has not been sorted into a meaningful order 3 how can one tell that the image is more likely to be a sheep than a train? -- Regards, =dn From mk1853387 at gmail.com Wed Jul 31 06:27:07 2024 From: mk1853387 at gmail.com (marc nicole) Date: Wed, 31 Jul 2024 12:27:07 +0200 Subject: Predicting an object over an pretrained model is not working In-Reply-To: <339c86d5-8cb5-4995-b5ba-12f88d71a107@DancesWithMice.info> References: <339c86d5-8cb5-4995-b5ba-12f88d71a107@DancesWithMice.info> Message-ID: I suppose the meaning of those numbers comes from this line predicts_dict[class_name].append([int(xmin), int(ymin), int(xmax), int(ymax), P[index]]) as well as the yolo inference call. But i was expecting zeros for all classes except smallball. Because the image only shows that, and that a train and a sheep wont have any target position or any probability whatsoever in the image weirdobject.jpg On Wed, 31 Jul 2024, 00:19 dn via Python-list, wrote: > On 31/07/24 06:18, marc nicole via Python-list wrote: > > Hello all, > > > > I want to predict an object by given as input an image and want to have > my > > model be able to predict the label. I have trained a model using > tensorflow > > based on annotated database where the target object to predict was added > to > > the pretrained model. the code I am using is the following where I set > the > > target object image as input and want to have the prediction output: > > ... > > > > WHile I expect only the dict to contain the small_ball key > > > How's that is possible? where's the prediction output?How to fix the > code? > > > To save us lots of reading and study to be able to help you, please advise: > > 1 what are the meanings of all these numbers? > > > 'sheep': [[233.0, 92.0, 448.0, -103.0, > >> 5.3531270027160645], [167.0, 509.0, 209.0, 101.0, 4.947688579559326], > >> [0.0, 0.0, 448.0, 431.0, 3.393721580505371]] > > 2 (assuming it hasn't) why the dict has not been sorted into a > meaningful order > > 3 how can one tell that the image is more likely to be a sheep than a > train? > > -- > Regards, > =dn > -- > https://mail.python.org/mailman/listinfo/python-list > From grant.b.edwards at gmail.com Wed Jul 31 09:58:12 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 31 Jul 2024 09:58:12 -0400 (EDT) Subject: Predicting an object over an pretrained model is not working References: <339c86d5-8cb5-4995-b5ba-12f88d71a107@DancesWithMice.info> Message-ID: <4WYtwN4HnMznVHX@mail.python.org> On 2024-07-31, marc nicole via Python-list wrote: > I suppose the meaning of those numbers comes from this line > predicts_dict[class_name].append([int(xmin), int(ymin), int(xmax), > int(ymax), P[index]]) as well as the yolo inference call. But i was > expecting zeros for all classes except smallball. That's not how machine learning and object recognition works. > Because the image only shows that, You know that. The machine doesn't. > and that a train and a sheep wont have any target position or any > probability whatsoever in the image weirdobject.jpg That depends on the training data and how the model works. You should probably do some reading on neural networks, machine learning, and pattern/object recognition. You appear to be trying to use tools without understanding what they do or how they work. -- Grant From mk1853387 at gmail.com Wed Jul 31 15:59:58 2024 From: mk1853387 at gmail.com (marc nicole) Date: Wed, 31 Jul 2024 21:59:58 +0200 Subject: Predicting an object over an pretrained model is not working In-Reply-To: <4WYtwN4HnMznVHX@mail.python.org> References: <339c86d5-8cb5-4995-b5ba-12f88d71a107@DancesWithMice.info> <4WYtwN4HnMznVHX@mail.python.org> Message-ID: You invitation to read on machine is not helping, if you wanna enlighten us on this specific case otherwise pls spare me such comments which i know On Wed, 31 Jul 2024, 16:00 Grant Edwards via Python-list, < python-list at python.org> wrote: > On 2024-07-31, marc nicole via Python-list wrote: > > > I suppose the meaning of those numbers comes from this line > > predicts_dict[class_name].append([int(xmin), int(ymin), int(xmax), > > int(ymax), P[index]]) as well as the yolo inference call. But i was > > expecting zeros for all classes except smallball. > > That's not how machine learning and object recognition works. > > > Because the image only shows that, > > You know that. The machine doesn't. > > > and that a train and a sheep wont have any target position or any > > probability whatsoever in the image weirdobject.jpg > > That depends on the training data and how the model works. > > You should probably do some reading on neural networks, machine > learning, and pattern/object recognition. You appear to be trying to > use tools without understanding what they do or how they work. > > -- > Grant > -- > https://mail.python.org/mailman/listinfo/python-list >