From alexkleider at gmail.com  Mon Jun  6 16:04:32 2022
From: alexkleider at gmail.com (Alex Kleider)
Date: Mon, 6 Jun 2022 13:04:32 -0700
Subject: [Tutor] unicode question
Message-ID: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>

I've been playing around with unicode a bit and found that the
following code doesn't behave as I might have expected:

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("\N{middle dot}")
?
>>>
>>> middle_dot = '0140', '013F', '00B7', '2027'
>>> ucode = ['\\u' + dot for dot in middle_dot]
>>> for dot in ucode:
...     print(dot)
...
\u0140
\u013F
\u00B7
\u2027
>>> print("\u0140")
?
>>> print("\u013f")
?
>>> print("\u00b7")  # the one I want
?
>>> print("\u2027")
?
>>>

I was expecting the for loop to output the same as the last four print
statements but, alas, not so.
Comments would be welcome.
Sincerely,
Alex

-- 
alex at kleider.ca  (sent from my current gizmo)

From __peter__ at web.de  Mon Jun  6 17:56:29 2022
From: __peter__ at web.de (Peter Otten)
Date: Mon, 6 Jun 2022 23:56:29 +0200
Subject: [Tutor] unicode question
In-Reply-To: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
References: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
Message-ID: <d44c6043-55d0-6c80-45ca-46f414bd9622@web.de>

On 06/06/2022 22:04, Alex Kleider wrote:
> I've been playing around with unicode a bit and found that the
> following code doesn't behave as I might have expected:
>
> Python 3.9.2 (default, Feb 28 2021, 17:03:44)
> [GCC 10.2.1 20210110] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> print("\N{middle dot}")
> ?
>>>>
>>>> middle_dot = '0140', '013F', '00B7', '2027'
>>>> ucode = ['\\u' + dot for dot in middle_dot]
>>>> for dot in ucode:
> ...     print(dot)
> ...
> \u0140
> \u013F
> \u00B7
> \u2027
>>>> print("\u0140")
> ?
>>>> print("\u013f")
> ?
>>>> print("\u00b7")  # the one I want
> ?
>>>> print("\u2027")
> ?
>>>>
>
> I was expecting the for loop to output the same as the last four print
> statements but, alas, not so.

"\\u" is a string containing the backslash followed by a "u" -- and that
won't change when you concatenate another string like "0140".

The easiest way to realize the loop would be to use integers:

 >>> for i in 0x140, 0x13f: print(chr(i))

?
?

The obvious way when you want to start with strings is

 >>> for c in "0140", "013f":
     print(eval(f"'\\u{c}'"))  # dangerous, may execute arbitrary code

?
?

with the safe alternative

 >>> for c in "0140", "013f":
	print(codecs.decode(f"\\u{c}", "unicode-escape"))


?
?

From alan.gauld at yahoo.co.uk  Mon Jun  6 18:22:57 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 6 Jun 2022 23:22:57 +0100
Subject: [Tutor] unicode question
In-Reply-To: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
References: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
Message-ID: <t7luo2$sim$1@ciao.gmane.io>

On 06/06/2022 21:04, Alex Kleider wrote:

>>>> middle_dot = '0140', '013F', '00B7', '2027'
>>>> ucode = ['\\u' + dot for dot in middle_dot]

This creates a list of literal strings like

"\u0140",....

Which is identical to writing:

ucode = ["\\u0140",...]

So you have the \u as characters within the string
which is not what you want.

Instead you want to get the character values from
the hex values in middle dot which you can do using
chr() and int().

for dot in middle_dot:
   print(chr(int(dot,16)))

However it would be easier just to store the hex values directly:

middle_dot = [0x0140, 0x013F, 0x00B7, 0x2027)
for dot in middle_dot:
    print(chr(dot))

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From mats at wichmann.us  Mon Jun  6 19:30:53 2022
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 6 Jun 2022 17:30:53 -0600
Subject: [Tutor] unicode question
In-Reply-To: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
References: <CAMCEyD6oRFsw5mcq+hbyKKaSe00pxm3p6Y3PRLaC7-VEEP1mmA@mail.gmail.com>
Message-ID: <56dfac24-29a1-5ada-cd90-2c65f368d98b@wichmann.us>

On 6/6/22 14:04, Alex Kleider wrote:
> I've been playing around with unicode a bit and found that the
> following code doesn't behave as I might have expected:
> 
> Python 3.9.2 (default, Feb 28 2021, 17:03:44)
> [GCC 10.2.1 20210110] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> print("\N{middle dot}")
> ?
>>>>
>>>> middle_dot = '0140', '013F', '00B7', '2027'

Why not directly enter these in a usable form?

>>> middle_dot = "\N{LATIN SMALL LETTER L WITH MIDDLE DOT}", "\N{LATIN
CAPITAL LETTER L WITH MIDDLE DOT}"
>>> for n in middle_dot:
...     print(n)
...
...
?
?


(too lazy to go look up the other two, whatever they may be)


From drgauravshukla at gmail.com  Tue Jun  7 06:33:09 2022
From: drgauravshukla at gmail.com (Gaurav Shukla)
Date: Tue, 7 Jun 2022 16:03:09 +0530
Subject: [Tutor] Gaussian to image
Message-ID: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>

Hello,
Is there a way to convert Gaussian curves to a greyscale image simulating a
CCD image?
So basically my Gauss has an x-value and y-value with multiple Gaussian's,
which one generally get from a CCD/sCMOS experimentally.

I have generated a Gaussian spectrum (with four peaks) based on atomic
physics calculations, I would then try to read with another code that would
read the image and give back the Gaussian spectrum.

I am very new so any help would be highly appreciated.

Thanks
DG

From wlfraed at ix.netcom.com  Tue Jun  7 17:04:22 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Tue, 07 Jun 2022 17:04:22 -0400
Subject: [Tutor] Gaussian to image
References: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>
Message-ID: <l0fv9hd15jepe9i1cdfs4912kgfvo9l6pk@4ax.com>

On Tue, 7 Jun 2022 16:03:09 +0530, Gaurav Shukla <drgauravshukla at gmail.com>
declaimed the following:

>Hello,
>Is there a way to convert Gaussian curves to a greyscale image simulating a
>CCD image?
>So basically my Gauss has an x-value and y-value with multiple Gaussian's,
>which one generally get from a CCD/sCMOS experimentally.
>

	I have no idea of exactly what that sentence is describing... "x-value
and y-value" to me imply ONE POINT on a Cartesian plane, so "multiple
Gaussian's" is a meaningless term.

	A 2D array/matrix (x-axis, y-axis) in which the each point has a value
representing the grey-level resulting from (possibly overlapping) gaussian
data, OTOH, I can understand.

	However, in general, have you studied the capabilities of the
MatPlotLib package? https://matplotlib.org/


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From alan.gauld at yahoo.co.uk  Tue Jun  7 18:35:22 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 7 Jun 2022 23:35:22 +0100
Subject: [Tutor] Gaussian to image
In-Reply-To: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>
References: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>
Message-ID: <t7ojra$ssf$1@ciao.gmane.io>

On 07/06/2022 11:33, Gaurav Shukla wrote:
> Hello,
> Is there a way to convert Gaussian curves to a greyscale image simulating a
> CCD image?

Probably, but its not really a Pthon question but a mathematical one.
You may be better off asking on one of the Python scientific computing
forums. sciPy might be a good start.


> So basically my Gauss has an x-value and y-value with multiple Gaussian's,
> which one generally get from a CCD/sCMOS experimentally.
> 
> I have generated a Gaussian spectrum (with four peaks) based on atomic
> physics calculations, I would then try to read with another code that would
> read the image and give back the Gaussian spectrum.

That is all very domain specific. But this is a general
programming forum concerned with the Python language and
its standard library.

It sounds like your request is very domain specific so
either you need to find a forum for that domain(see above)
or you have to go back to basics and exolain what those
2 paragraphs mean in simple terms a layman can understand.

> I am very new so any help would be highly appreciated.

We are happy to help with any Python programming questions.
But how you apply math to this specific question is a bit
beyond our remit. If you are very lucky one or two of our
members will recognise what you are saying, but you are
much more likely to find that on the Scipy fora


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From PythonList at DancesWithMice.info  Tue Jun  7 18:56:40 2022
From: PythonList at DancesWithMice.info (dn)
Date: Wed, 8 Jun 2022 10:56:40 +1200
Subject: [Tutor] Gaussian to image
In-Reply-To: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>
References: <CACQHu+4-pQLw73syhfkU5ZXUTot_rfnhTE9yUtyBNWSv-oo3nQ@mail.gmail.com>
Message-ID: <e8c3fbb3-860b-0be8-c855-3995c63483b5@DancesWithMice.info>

On 07/06/2022 22.33, Gaurav Shukla wrote:
> Hello,
> Is there a way to convert Gaussian curves to a greyscale image simulating a
> CCD image?
> So basically my Gauss has an x-value and y-value with multiple Gaussian's,
> which one generally get from a CCD/sCMOS experimentally.
> 
> I have generated a Gaussian spectrum (with four peaks) based on atomic
> physics calculations, I would then try to read with another code that would
> read the image and give back the Gaussian spectrum.
> 
> I am very new so any help would be highly appreciated.

Might OpenCV help?
(https://opencv.org/opencv-python-is-now-an-official-opencv-project/ and
plenty of tutorials on the web)
-- 
Regards,
=dn

From aliyan.navaid at gmail.com  Fri Jun 10 08:59:25 2022
From: aliyan.navaid at gmail.com (Aliyan Navaid)
Date: Fri, 10 Jun 2022 17:59:25 +0500
Subject: [Tutor] Learning Object Oriented Programming
Message-ID: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>

   Hello,

   I aspire to learn AI with python and ?learned the basics of python such as
   datatypes, iteration, selection.

   However now I?m stuck as there is so much in the field of AI, what should
   my next move be? Learning OOP? Libraries such as SciPy? Tensor flow?
   Statistics?

   Thanks in advance.

From leamhall at gmail.com  Fri Jun 10 15:51:45 2022
From: leamhall at gmail.com (Leam Hall)
Date: Fri, 10 Jun 2022 14:51:45 -0500
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
Message-ID: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>

Learning Python's object system will probably help, since most of your other material will assume you know it. As you said, the AI field is huge. Maybe mention one or two things that really excite you and see if folks can make suggestions for those specific things.


On 6/10/22 07:59, Aliyan Navaid wrote:
>     Hello,
> 
>     I aspire to learn AI with python and ?learned the basics of python such as
>     datatypes, iteration, selection.
> 
>     However now I?m stuck as there is so much in the field of AI, what should
>     my next move be? Learning OOP? Libraries such as SciPy? Tensor flow?
>     Statistics?
> 
>     Thanks in advance.

-- 
Automation Engineer        (reuel.net/resume)
Scribe: The Domici War     (domiciwar.net)
General Ne'er-do-well      (github.com/LeamHall)

From aliyan.navaid at gmail.com  Sat Jun 11 04:40:01 2022
From: aliyan.navaid at gmail.com (Aliyan Navaid)
Date: Sat, 11 Jun 2022 13:40:01 +0500
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
 <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
Message-ID: <CAMsOQ-ETRaJWw8cX9=DpurZmQqMS3Pr1SFWyFtku-xvQVqiuvg@mail.gmail.com>

To be honest every AI project I've seen amazes me whether it is about
finding insights in data, NLP, or Image generation. However, could you list
down stuff (in order) that is essential to all these applications?

On Sat, 11 Jun 2022 at 00:53, Leam Hall <leamhall at gmail.com> wrote:

> Learning Python's object system will probably help, since most of your
> other material will assume you know it. As you said, the AI field is huge.
> Maybe mention one or two things that really excite you and see if folks can
> make suggestions for those specific things.
>
>
> On 6/10/22 07:59, Aliyan Navaid wrote:
> >     Hello,
> >
> >     I aspire to learn AI with python and  learned the basics of python
> such as
> >     datatypes, iteration, selection.
> >
> >     However now I?m stuck as there is so much in the field of AI, what
> should
> >     my next move be? Learning OOP? Libraries such as SciPy? Tensor flow?
> >     Statistics?
> >
> >     Thanks in advance.
>
> --
> Automation Engineer        (reuel.net/resume)
> Scribe: The Domici War     (domiciwar.net)
> General Ne'er-do-well      (github.com/LeamHall)
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From avi.e.gross at gmail.com  Fri Jun 10 16:52:48 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Fri, 10 Jun 2022 16:52:48 -0400
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
 <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
Message-ID: <00f501d87d0c$0bda8720$238f9560$@gmail.com>

My two cents is that the question is way too broad and this is not the place to discuss it.

[Most people here who are focused on learning the basics of python probably should skip this message.]

The subject was about Object Oriented Programming. The questions asked are NOT really.

They are about an assortment of modules that internally use all kinds of techniques that do often do create and manage a series of objects and functions and to use them the user may have to use Object Oriented techniques and often other things like functional programming techniques. 

I have studied base Python a dozen or more ways to gain some fluency and only then did I study the modules you mention and many others. You may be able to use some much better if you learn incrementally. You may need to study the numpy module that is heavily used by everything including YOUR code that has to put data into a form that can be processed well. You may want to study the pandas module that sort of extends numpy.

After that, it depends on what you want to do with AI. I am not sure you need SciPy for many things as much as Scikit-learn. As for TensorFlow, that is doing things at a relatively low level and many others have built on top of it some ways to do things in possibly simpler and more powerful ways such as Keras or Theano and others, albeit some take very different paths with some not feeling natural unless you understand many styles of python programming. 

But this is NOT the place for tutoring about specific and esoteric questions and as you have not asked such a question, I will not talk about this more here. There are oodles of resources on-line and in books that you can consult and so much depends on your background. AI is a vast subject and getting vaster and you may actually be more interested in topics like Machine Learning and end up working in a place where others use specific tools among the many available.

Python as a language is just one of many tools. It is commonly used for the general purposes mentioned but some places use others.

Good Luck.


-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of Leam Hall
Sent: Friday, June 10, 2022 3:52 PM
To: tutor at python.org
Subject: Re: [Tutor] Learning Object Oriented Programming

Learning Python's object system will probably help, since most of your other material will assume you know it. As you said, the AI field is huge. Maybe mention one or two things that really excite you and see if folks can make suggestions for those specific things.


On 6/10/22 07:59, Aliyan Navaid wrote:
>     Hello,
> 
>     I aspire to learn AI with python and  learned the basics of python such as
>     datatypes, iteration, selection.
> 
>     However now I?m stuck as there is so much in the field of AI, what should
>     my next move be? Learning OOP? Libraries such as SciPy? Tensor flow?
>     Statistics?
> 
>     Thanks in advance.

-- 
Automation Engineer        (reuel.net/resume)
Scribe: The Domici War     (domiciwar.net)
General Ne'er-do-well      (github.com/LeamHall)
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From alan.gauld at yahoo.co.uk  Sat Jun 11 07:55:44 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 11 Jun 2022 12:55:44 +0100
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <CAMsOQ-ETRaJWw8cX9=DpurZmQqMS3Pr1SFWyFtku-xvQVqiuvg@mail.gmail.com>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
 <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
 <CAMsOQ-ETRaJWw8cX9=DpurZmQqMS3Pr1SFWyFtku-xvQVqiuvg@mail.gmail.com>
Message-ID: <t81vs1$h46$1@ciao.gmane.io>

On 11/06/2022 09:40, Aliyan Navaid wrote:
> To be honest every AI project I've seen amazes me whether it is about
> finding insights in data, NLP, or Image generation. However, could you list
> down stuff (in order) that is essential to all these applications?
> 

I see Avi has already posted a reply suggesting you won't get your
answes here. That is true in terms of the ultimate and specific
answers to your question however I do think its a fair question
for a beginner to ask here. The reason is, and your question and
follow-up suggest it is true of you too, that most beginners see
an interesting field but have no idea of the complexity involved.
In your case AI is a whole sub genre of computing with many
specialisms within it. Until you know which specific sub-branch
you want to explore it is impossible to give specific advice.

What we can say is a that a good knowledge of general programming
techniques will never go amiss and Functional Programming (which
is not the same thing as just writing/using functions!) and the
use of classes and objects are both worth studying. I don't think
you need to study the full OOP paradigm since most AI projects
seem to be more likely to be functional in nature with just a
few classes or objects used as data carriers. This list and
other python resources are certainly able to help with that.

Other areas of general programming that may be worth considering
include data storage(RDBMS, NO-SQL, and other "Big Data" solutions)
as well as generic algorithms for searching/sorting etc. Wikipedia
is probably a good starting place for these areas.

Once you get into AI domain specific subjects you will probably
find domain-specific fora that are better placed to help.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avi.e.gross at gmail.com  Sat Jun 11 17:42:06 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Sat, 11 Jun 2022 17:42:06 -0400
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <t81vs1$h46$1@ciao.gmane.io>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
 <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
 <CAMsOQ-ETRaJWw8cX9=DpurZmQqMS3Pr1SFWyFtku-xvQVqiuvg@mail.gmail.com>
 <t81vs1$h46$1@ciao.gmane.io>
Message-ID: <00e301d87ddc$191540d0$4b3fc270$@gmail.com>

Alan,

My first post was moderated and I have not received a copy here showing me
it was shared. My main point was not that some aspects could not be
discussed here but that others would be better done elsewhere or 1:1 with
someone as this forum is dedicated for other purposes. I am adding or
expanding on some of what you wrote as I have no disagreement.

The part I found relevant was asking what aspects of Python the language to
focus on for some purpose, albeit I find such questions hard to answer.

Should you learn all about using sets in Python? Some people never find a
need to use them and many languages have no such construct. But not learning
them may confuse you later when you encounter what to you looks like a very
badly formed dictionary that seems to be missing values, or is it keys?

AI is not only a very broad field but a moving target that also keeps
evolving. If you are interested in the AI I was studying while working on a
Ph.D. in AI at Rutgers in the early 80's, it is barely recognizable and
definitely did not use Python. Much of what was seen as new and interesting
ended up migrating out from the umbrella of AI into other aspects of
computer science and has become so commonplace that it is not considered
challenging any more. Other aspects have fallen out of favor or been shown
to not be a great approach.

I have revisited many aspects of what is now AI in recent years and am
especially focused on many kinds of statistical methods that overlap AI and
I Machine Learning.  Much of that is done using Python extensions but I have
done much work you would consider to be aspects of AI in other languages
such as R or stand-alone applications. 

So the question here struck me as having several parts and that the person
asking was either not very clear in their mind and sort of hoping to find
some direction to get started. And, of course, sometimes it may simply be
that they are just not that proficient yet in English.

So if the question is whether learning a bit about Object-Oriented
techniques in Python if your plan is to study AI, then the answer might be
YES. But only up to a point. No need to be an expert at first.

Many aspects of AI are mostly done using modules and packages others have
already written and often tested. They come with pre-made objects that often
are usable as black boxes with details well hidden within and the manuals
tell you how to access or modify them, albeit good ones often hide much of
that too and provide you with all the interfaces you normally need. Of
course if you later want to tweak some things, you may need more, let alone
if you want to share your own creations.

Consider one of many scenarios in the scikit-learn module. You may want to
do SOMETHING (and I am being abstract) that has variations.

Typically you ask for a new object to be created that encapsulates what you
want to do. You call code like:

MyProblemSolver = doSomething(initializers)

You now have an object which has used one of the initializers defined within
that you do not need to see anything unless you feed it something wrong. It
can set  all kinds of parameters,  maybe check validity, maybe creates
accessory objects within and who knows what.

You may also tweak it using methods supplied in additional lines of code
like:

MyProblemSolver.dosomething(...)

I mean objectName.fit(...), objectName.transform(...),
objectName.predict(...) and so on.

One key part is giving it the data you want to use. For that, you may end up
calling it with a method you hand something to that can range from a python
list or some exotic object like a numpy Series or pandas DataFrame and so
on.

There are quite a few such object types  used for various purposes and you
can make many of them and tweak each and compare results and so on.

But you can use many of these tools without having the slightest idea what
an object is. A well designed tool may be easy to use by setting reasonable
defaults so if you want one of many other possible tweaks, like adjusting
the level of an alpha variable or choosing a different algorithm for some
part of a transaction by name or even supplying your own function for use in
comparing two things internally, you can tweak it quite a bit and even make
your own customizations. Beginners rarely need to.

As I mentioned, if you want to do something with some Machine Learning such
as a sort of neural network, a beginner is often better off skipping
learning powerful multi-purpose packages like TensorFlow and going straight
to Keras which is built on top of TensorFlow and pandas and other things and
largely removes you even further from seeing the endless details beneath BUT
may let you dip lower if you wish.

Much depends on whether you are just interested in the topic, are a student
hoping to major in something, or even looking at specific jobs. Many skills
are not worthwhile to spend lots of time on except as a hobby.

And, as noted, some of the modules written that encapsulate TensorFlow have
a rather different but still pythonic aspect as what they do is less
object-oriented and more function-oriented as you add layers by
encapsulating functions within functions.  To use those you need to focus on
other aspects of python.

But here is an important point. Generally you do NOT do machine learning or
other forms of AI alone. You often do lots of things in base python or other
modules to read in the data, clean it up or do calculations on it, perhaps
present graphics about it, and much more. Then you do one or more analyses
as described above and often do more calculations on the various parts
created in the process such as combining several and choosing which model
made better results on your test cases and perhaps more analyses and graphs
and ...

So your overall knowledge of other aspects of python and the many modules
people like for these tasks is important. I personally like to do some
things using other languages and tools and often do programming using
environments that create documents (as in a PDF or HTML or WORD documents)
that include marking up all kinds of text as well as including snippets of
python and R code that inter-operate with each other on the same data where
each does what it is good at or provides what I like. This is what happens
not at the beginning of a learning process but way later.  You sound like
you need some baby steps headed in the right direction. 

So, as others have said, learn enough about python without worrying if you
have mastered ALL the many ways to create formatted text and so on. I
suggest that much of AI is data intensive and that learning numpy decently
(but not expertly) should come next. Depending on your future needs, pandas
would come next, or might be studied only if it ever is needed. You can
always come back and learn missing things such as how to use sets. Man/most
languages have no such thing built in and yet people make do using things
like vectors of items along with abilities to make the contents unique() or
do various union() and intersection() and setdiff() operations OR creating
your own object-like creations or using packages from others. 

But you do not need to do this linearly. You can use later stages to help
guide you by seeing what code they discuss that you have not understood. You
can fill in the gaps. And you would be amazed how much (along with
misleading nonsense) you can pick up with a web search or two if you include
things like the name of the language and module as in "python numpy remove
duplicates" or whatever you can phrase well enough.  Frankly, it can be a
mistake to study python in a multi-volume way when the vast majority of what
you learn is overkill. An intro book or two and maybe something intermediate
but focused such as found using "book machine learning in python" or one
about graphics and python may let you make progress.

Now if you have specific questions about some code you read about use of
objects and want to understand, this could be the place, but not if you have
read nothing and want one on one attention to learn as that may require a
real full-time dedicated person as a tutor.

Again, good luck. Questions deeper than basic python, though, often have a
better place, or even better, find resources you can read and practice with
on your own as such learning can be longer-lasting. 


-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Alan Gauld via Tutor
Sent: Saturday, June 11, 2022 7:56 AM
To: tutor at python.org
Subject: Re: [Tutor] Learning Object Oriented Programming

On 11/06/2022 09:40, Aliyan Navaid wrote:
> To be honest every AI project I've seen amazes me whether it is about 
> finding insights in data, NLP, or Image generation. However, could you 
> list down stuff (in order) that is essential to all these applications?
> 

I see Avi has already posted a reply suggesting you won't get your answes
here. That is true in terms of the ultimate and specific answers to your
question however I do think its a fair question for a beginner to ask here.
The reason is, and your question and follow-up suggest it is true of you
too, that most beginners see an interesting field but have no idea of the
complexity involved.
In your case AI is a whole sub genre of computing with many specialisms
within it. Until you know which specific sub-branch you want to explore it
is impossible to give specific advice.

What we can say is a that a good knowledge of general programming techniques
will never go amiss and Functional Programming (which is not the same thing
as just writing/using functions!) and the use of classes and objects are
both worth studying. I don't think you need to study the full OOP paradigm
since most AI projects seem to be more likely to be functional in nature
with just a few classes or objects used as data carriers. This list and
other python resources are certainly able to help with that.

Other areas of general programming that may be worth considering include
data storage(RDBMS, NO-SQL, and other "Big Data" solutions) as well as
generic algorithms for searching/sorting etc. Wikipedia is probably a good
starting place for these areas.

Once you get into AI domain specific subjects you will probably find
domain-specific fora that are better placed to help.


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From rakesh7biswas at gmail.com  Sat Jun 11 21:41:49 2022
From: rakesh7biswas at gmail.com (Rakesh Biswas)
Date: Sun, 12 Jun 2022 07:11:49 +0530
Subject: [Tutor] Learning Object Oriented Programming
In-Reply-To: <00e301d87ddc$191540d0$4b3fc270$@gmail.com>
References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol>
 <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com>
 <CAMsOQ-ETRaJWw8cX9=DpurZmQqMS3Pr1SFWyFtku-xvQVqiuvg@mail.gmail.com>
 <t81vs1$h46$1@ciao.gmane.io> <00e301d87ddc$191540d0$4b3fc270$@gmail.com>
Message-ID: <CADQZ6X9Y+zCkVqvFdK4+iyLxWpRTmxYNvXccNcNbpRrxVxZD=Q@mail.gmail.com>

It was shared to this list and it was a great email but this one was even
better.

best,

rb


On Sun, Jun 12, 2022, 4:14 AM <avi.e.gross at gmail.com> wrote:

> Alan,
>
> My first post was moderated and I have not received a copy here showing me
> it was shared. My main point was not that some aspects could not be
> discussed here but that others would be better done elsewhere or 1:1 with
> someone as this forum is dedicated for other purposes. I am adding or
> expanding on some of what you wrote as I have no disagreement.
>
> The part I found relevant was asking what aspects of Python the language to
> focus on for some purpose, albeit I find such questions hard to answer.
>
> Should you learn all about using sets in Python? Some people never find a
> need to use them and many languages have no such construct. But not
> learning
> them may confuse you later when you encounter what to you looks like a very
> badly formed dictionary that seems to be missing values, or is it keys?
>
> AI is not only a very broad field but a moving target that also keeps
> evolving. If you are interested in the AI I was studying while working on a
> Ph.D. in AI at Rutgers in the early 80's, it is barely recognizable and
> definitely did not use Python. Much of what was seen as new and interesting
> ended up migrating out from the umbrella of AI into other aspects of
> computer science and has become so commonplace that it is not considered
> challenging any more. Other aspects have fallen out of favor or been shown
> to not be a great approach.
>
> I have revisited many aspects of what is now AI in recent years and am
> especially focused on many kinds of statistical methods that overlap AI and
> I Machine Learning.  Much of that is done using Python extensions but I
> have
> done much work you would consider to be aspects of AI in other languages
> such as R or stand-alone applications.
>
> So the question here struck me as having several parts and that the person
> asking was either not very clear in their mind and sort of hoping to find
> some direction to get started. And, of course, sometimes it may simply be
> that they are just not that proficient yet in English.
>
> So if the question is whether learning a bit about Object-Oriented
> techniques in Python if your plan is to study AI, then the answer might be
> YES. But only up to a point. No need to be an expert at first.
>
> Many aspects of AI are mostly done using modules and packages others have
> already written and often tested. They come with pre-made objects that
> often
> are usable as black boxes with details well hidden within and the manuals
> tell you how to access or modify them, albeit good ones often hide much of
> that too and provide you with all the interfaces you normally need. Of
> course if you later want to tweak some things, you may need more, let alone
> if you want to share your own creations.
>
> Consider one of many scenarios in the scikit-learn module. You may want to
> do SOMETHING (and I am being abstract) that has variations.
>
> Typically you ask for a new object to be created that encapsulates what you
> want to do. You call code like:
>
> MyProblemSolver = doSomething(initializers)
>
> You now have an object which has used one of the initializers defined
> within
> that you do not need to see anything unless you feed it something wrong. It
> can set  all kinds of parameters,  maybe check validity, maybe creates
> accessory objects within and who knows what.
>
> You may also tweak it using methods supplied in additional lines of code
> like:
>
> MyProblemSolver.dosomething(...)
>
> I mean objectName.fit(...), objectName.transform(...),
> objectName.predict(...) and so on.
>
> One key part is giving it the data you want to use. For that, you may end
> up
> calling it with a method you hand something to that can range from a python
> list or some exotic object like a numpy Series or pandas DataFrame and so
> on.
>
> There are quite a few such object types  used for various purposes and you
> can make many of them and tweak each and compare results and so on.
>
> But you can use many of these tools without having the slightest idea what
> an object is. A well designed tool may be easy to use by setting reasonable
> defaults so if you want one of many other possible tweaks, like adjusting
> the level of an alpha variable or choosing a different algorithm for some
> part of a transaction by name or even supplying your own function for use
> in
> comparing two things internally, you can tweak it quite a bit and even make
> your own customizations. Beginners rarely need to.
>
> As I mentioned, if you want to do something with some Machine Learning such
> as a sort of neural network, a beginner is often better off skipping
> learning powerful multi-purpose packages like TensorFlow and going straight
> to Keras which is built on top of TensorFlow and pandas and other things
> and
> largely removes you even further from seeing the endless details beneath
> BUT
> may let you dip lower if you wish.
>
> Much depends on whether you are just interested in the topic, are a student
> hoping to major in something, or even looking at specific jobs. Many skills
> are not worthwhile to spend lots of time on except as a hobby.
>
> And, as noted, some of the modules written that encapsulate TensorFlow have
> a rather different but still pythonic aspect as what they do is less
> object-oriented and more function-oriented as you add layers by
> encapsulating functions within functions.  To use those you need to focus
> on
> other aspects of python.
>
> But here is an important point. Generally you do NOT do machine learning or
> other forms of AI alone. You often do lots of things in base python or
> other
> modules to read in the data, clean it up or do calculations on it, perhaps
> present graphics about it, and much more. Then you do one or more analyses
> as described above and often do more calculations on the various parts
> created in the process such as combining several and choosing which model
> made better results on your test cases and perhaps more analyses and graphs
> and ...
>
> So your overall knowledge of other aspects of python and the many modules
> people like for these tasks is important. I personally like to do some
> things using other languages and tools and often do programming using
> environments that create documents (as in a PDF or HTML or WORD documents)
> that include marking up all kinds of text as well as including snippets of
> python and R code that inter-operate with each other on the same data where
> each does what it is good at or provides what I like. This is what happens
> not at the beginning of a learning process but way later.  You sound like
> you need some baby steps headed in the right direction.
>
> So, as others have said, learn enough about python without worrying if you
> have mastered ALL the many ways to create formatted text and so on. I
> suggest that much of AI is data intensive and that learning numpy decently
> (but not expertly) should come next. Depending on your future needs, pandas
> would come next, or might be studied only if it ever is needed. You can
> always come back and learn missing things such as how to use sets. Man/most
> languages have no such thing built in and yet people make do using things
> like vectors of items along with abilities to make the contents unique() or
> do various union() and intersection() and setdiff() operations OR creating
> your own object-like creations or using packages from others.
>
> But you do not need to do this linearly. You can use later stages to help
> guide you by seeing what code they discuss that you have not understood.
> You
> can fill in the gaps. And you would be amazed how much (along with
> misleading nonsense) you can pick up with a web search or two if you
> include
> things like the name of the language and module as in "python numpy remove
> duplicates" or whatever you can phrase well enough.  Frankly, it can be a
> mistake to study python in a multi-volume way when the vast majority of
> what
> you learn is overkill. An intro book or two and maybe something
> intermediate
> but focused such as found using "book machine learning in python" or one
> about graphics and python may let you make progress.
>
> Now if you have specific questions about some code you read about use of
> objects and want to understand, this could be the place, but not if you
> have
> read nothing and want one on one attention to learn as that may require a
> real full-time dedicated person as a tutor.
>
> Again, good luck. Questions deeper than basic python, though, often have a
> better place, or even better, find resources you can read and practice with
> on your own as such learning can be longer-lasting.
>
>
> -----Original Message-----
> From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
> Alan Gauld via Tutor
> Sent: Saturday, June 11, 2022 7:56 AM
> To: tutor at python.org
> Subject: Re: [Tutor] Learning Object Oriented Programming
>
> On 11/06/2022 09:40, Aliyan Navaid wrote:
> > To be honest every AI project I've seen amazes me whether it is about
> > finding insights in data, NLP, or Image generation. However, could you
> > list down stuff (in order) that is essential to all these applications?
> >
>
> I see Avi has already posted a reply suggesting you won't get your answes
> here. That is true in terms of the ultimate and specific answers to your
> question however I do think its a fair question for a beginner to ask here.
> The reason is, and your question and follow-up suggest it is true of you
> too, that most beginners see an interesting field but have no idea of the
> complexity involved.
> In your case AI is a whole sub genre of computing with many specialisms
> within it. Until you know which specific sub-branch you want to explore it
> is impossible to give specific advice.
>
> What we can say is a that a good knowledge of general programming
> techniques
> will never go amiss and Functional Programming (which is not the same thing
> as just writing/using functions!) and the use of classes and objects are
> both worth studying. I don't think you need to study the full OOP paradigm
> since most AI projects seem to be more likely to be functional in nature
> with just a few classes or objects used as data carriers. This list and
> other python resources are certainly able to help with that.
>
> Other areas of general programming that may be worth considering include
> data storage(RDBMS, NO-SQL, and other "Big Data" solutions) as well as
> generic algorithms for searching/sorting etc. Wikipedia is probably a good
> starting place for these areas.
>
> Once you get into AI domain specific subjects you will probably find
> domain-specific fora that are better placed to help.
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From anmol.ece at gmail.com  Sun Jun 12 00:10:40 2022
From: anmol.ece at gmail.com (Anmol Sharma)
Date: Sun, 12 Jun 2022 09:40:40 +0530
Subject: [Tutor] Issue with running iPython
Message-ID: <CA+ayJCNRoakgCzyr5ANgaUSR6Z2U5+umUg5xHfQ7wi3sjx+Vhw@mail.gmail.com>

I just installed python on win 10 and installed Anaconda distribution. I
get the following error while using ipython.Python error -
self._poll(timeout) RuntimeError: <overlapped.Overlapped object at <0>
still has pending operation at deallocation, process may crash
This error is coming without any code also. SO, if i just load ipython and
it taken me to ipython prompt, then If I just press "enter" multiple times,
it gives this error.

I also imported the necessary packages - import platform, asyncio if
platform.system() == 'Windows':
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

And then I just load ipython and without even writing any line of code, I
just press enter 2-3 times and I get this error -

(base) C:\Users\03987N744\Training_notebook>ipython -i --no-banner
enumerate_vs_range.py

In [1]:

In [1]:

In [1]:

Traceback (most recent call last): File
"C:\Users\Anaconda3\lib\asyncio\windows_events.py", line 439, in select
self._poll(timeout) RuntimeError: <_overlapped.Overlapped object at
0x0000022127E19DB0> still has pending operation at deallocation, the
process may crash

-- Also, this seems to be specific to *i*python and jupyter notebooks... It
does not occur on python....


Regards

Anmol Sharma

From alan.gauld at yahoo.co.uk  Sun Jun 12 07:04:46 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 12 Jun 2022 12:04:46 +0100
Subject: [Tutor] Issue with running iPython
In-Reply-To: <CA+ayJCNRoakgCzyr5ANgaUSR6Z2U5+umUg5xHfQ7wi3sjx+Vhw@mail.gmail.com>
References: <CA+ayJCNRoakgCzyr5ANgaUSR6Z2U5+umUg5xHfQ7wi3sjx+Vhw@mail.gmail.com>
Message-ID: <t84h8e$99v$1@ciao.gmane.io>

On 12/06/2022 05:10, Anmol Sharma wrote:

> And then I just load ipython and without even writing any line of code, I
> just press enter 2-3 times and I get this error -
> 
> (base) C:\Users\03987N744\Training_notebook>ipython -i --no-banner
> enumerate_vs_range.py

What happens if you start ipython without the enumerate_vs_range.py
file? Does it work OK then?

And what happens if you start it without the -i and --no_banner flags?
That might help isolate where the error is coming from.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From nathan-tech at hotmail.com  Tue Jun 14 01:31:56 2022
From: nathan-tech at hotmail.com (Nathan Smith)
Date: Tue, 14 Jun 2022 06:31:56 +0100
Subject: [Tutor] writing to an address in memory
Message-ID: <DB7PR07MB5093412A3B7595F01CE1C37BE4AA9@DB7PR07MB5093.eurprd07.prod.outlook.com>

Hi!


I'm working with a library which written in C and has a callback 
function for listening for data, basically.

Essentially any time it wants new data, it calls the callback with 
arguments:

callback(handle, buffer, length)

Where bufffer is a pointer to the buffer to write the data to.


I know how to read the data (using ctypes.from_address) but I don't know 
how to write data to it?

I've tried Googling, but I think my research is being hampered that I 
may not be using the correct termonology?

The library I am working with is:

http://www.un4seen.com/doc/#bass/STREAMPROC.html

-- 

Best Wishes,

Nathan Smith, BSC


My Website: https://nathantech.net



From wlfraed at ix.netcom.com  Tue Jun 14 11:37:35 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Tue, 14 Jun 2022 11:37:35 -0400
Subject: [Tutor] writing to an address in memory
References: <DB7PR07MB5093412A3B7595F01CE1C37BE4AA9@DB7PR07MB5093.eurprd07.prod.outlook.com>
Message-ID: <15ahahdcdq4imse4ab8l9052kq22546dem@4ax.com>

On Tue, 14 Jun 2022 06:31:56 +0100, Nathan Smith <nathan-tech at hotmail.com>
declaimed the following:

>Hi!
>
>
>I'm working with a library which written in C and has a callback 
>function for listening for data, basically.
>
>Essentially any time it wants new data, it calls the callback with 
>arguments:
>
>callback(handle, buffer, length)
>
>Where bufffer is a pointer to the buffer to write the data to.
>
>
>I know how to read the data (using ctypes.from_address) but I don't know 
>how to write data to it?

	Do
https://docs.python.org/3/library/ctypes.html#passing-pointers-or-passing-parameters-by-reference
https://docs.python.org/3/library/ctypes.html#ctypes.byref
https://docs.python.org/3/library/ctypes.html#callback-functions
provide any hints (caveat: I've not used ctypes, I just read documentation
<G>)



-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From phillor9 at gmail.com  Sat Jun 18 01:04:50 2022
From: phillor9 at gmail.com (Phil)
Date: Sat, 18 Jun 2022 15:04:50 +1000
Subject: [Tutor] Event loop logic question
Message-ID: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>

Thank you for reading this.

What I'm trying to achieve here is have the word 'stop' printed once 
whenever no keys are pressed and 'forward' printed once even when the up 
arrow key is held down.

I don't think this question needs any knowledge of Pygame Zero to solve 
and it's quite likely that I've? misunderstood the logic required. The 
following code is my most recent version which seems to work the first 
time through the loop but fails after subsequent 'up' key presses. 
Debugging the event loop is difficult.

Any hints or alternative methods will be, as always, greatly appreciated.

By the way, this project sends commands to a controller which is easily 
overwhelmed if too many commands are received per second.

import pgzrun


WIDTH = 20
HEIGHT = 20

direction = 'stop'
stop_flag = True
forward_flag = True

def on_key_down(key):
 ??? global direction

 ??? if key == keys.UP:
 ??????? print('up pressed')
 ??????? direction = 'forward'
 ??????? forward_flag = True
 ??????? print('forward_flag ', forward_flag)

 ??? elif key == keys.SPACE:
 ??????? direction = 'quit'

def on_key_up(key):
 ??? global direction

 ??? #print('stop')
 ??? direction = 'stop'
 ??? stop_flag = True

def update():
 ??? global direction, stop_flag, forward_flag

 ??? if direction == 'stop' and stop_flag:
 ??????? print('stop')
 ??????? stop_flag = False

 ??? elif direction == 'forward' and forward_flag:
 ??????? print('forward')
 ??????? forward_flag = False
 ??????? stop_flag = True

 ??? elif direction == 'quit':
 ??????? quit()

pgzrun.go()

-- 
Regards,
Phil


From PythonList at DancesWithMice.info  Sat Jun 18 01:14:26 2022
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 18 Jun 2022 17:14:26 +1200
Subject: [Tutor] Event loop logic question
In-Reply-To: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
Message-ID: <f277d92f-d46b-5081-d8ba-8990757292b8@DancesWithMice.info>

On 18/06/2022 17.04, Phil wrote:
> Thank you for reading this.
> 
> What I'm trying to achieve here is have the word 'stop' printed once
> whenever no keys are pressed and 'forward' printed once even when the up
> arrow key is held down.
> 
> I don't think this question needs any knowledge of Pygame Zero to solve
> and it's quite likely that I've? misunderstood the logic required. The
> following code is my most recent version which seems to work the first
> time through the loop but fails after subsequent 'up' key presses.
> Debugging the event loop is difficult.
> 
> Any hints or alternative methods will be, as always, greatly appreciated.
> 
> By the way, this project sends commands to a controller which is easily
> overwhelmed if too many commands are received per second.

You may like to study the games (start with snake) at
https://simplegametutorials.github.io/pygamezero/ to see how they do it.
-- 
Regards,
=dn

From phillor9 at gmail.com  Sat Jun 18 01:45:57 2022
From: phillor9 at gmail.com (Phil)
Date: Sat, 18 Jun 2022 15:45:57 +1000
Subject: [Tutor] Event loop logic question
In-Reply-To: <f277d92f-d46b-5081-d8ba-8990757292b8@DancesWithMice.info>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <f277d92f-d46b-5081-d8ba-8990757292b8@DancesWithMice.info>
Message-ID: <13dd409b-5240-0a03-6d22-883a63f5684a@gmail.com>


On 18/6/22 15:14, dn wrote:
>
> You may like to study the games (start with snake) at
> https://simplegametutorials.github.io/pygamezero/ to see how they do it.

Thanks dn. I had already looked through the examples but they didn't 
seem to relate to what I'm trying to achieve. I'll have another look.

-- 

Regards,
Phil


From alan.gauld at yahoo.co.uk  Sat Jun 18 06:05:10 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 18 Jun 2022 11:05:10 +0100
Subject: [Tutor] Event loop logic question
In-Reply-To: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
Message-ID: <t8k80n$dqc$1@ciao.gmane.io>

On 18/06/2022 06:04, Phil wrote:
> Thank you for reading this.
> 
> What I'm trying to achieve here is have the word 'stop' printed once 
> whenever no keys are pressed and 'forward' printed once even when the up 
> arrow key is held down.

I understand the second part, but not the first.
How do you determine that no keys are pressed - how you detect the
absence of an event? Usually there is some kind of dummy "idle"
event that you could use to trigger it, but I don't know your toolkit.
However you seem to be using the key_up event which will trigger
between every key press whioch is not, I think, what you want?

> I don't think this question needs any knowledge of Pygame Zero to solve

On the contrary, knowing what events are available from the toolkit
seems to be crucial to how you solve this. I can do it in tkinter (I
think!) but have no idea about PygameZero, which i'd not heard of till now!


> following code is my most recent version which seems to work the first 
> time through the loop but fails after subsequent 'up' key presses. 

Can you be more specific? What does "fail" mean? What does it actually
do? Does it print anything? Print too much(of what?) or crash? Is there
an error message?

> Debugging the event loop is difficult.

The usual way is to insert print satements into the event handlers or to
catch the event handlers in a debugger using breakpoints.

> direction = 'stop'
> stop_flag = True
> forward_flag = True
> 
> def on_key_down(key):
>  ??? global direction
> 
>  ??? if key == keys.UP:
>  ??????? print('up pressed')
>  ??????? direction = 'forward'
>  ??????? forward_flag = True
>  ??????? print('forward_flag ', forward_flag)

Its usual practice not to do UI changes(ie print) in the event
handlers but to do that in the draw/update/refresh method.
Just set the values in the handlers.

Another question - what should happen if another key other than
UP (or space) is pressed? Would you want to cancel the forward flag?

>  ??? elif key == keys.SPACE:
>  ??????? direction = 'quit'

Why set direction to quit? It's not really a direction.
Why not just quit right here? If you did want a farewell
message then fair enough, although I'd tend to use a
flag rather than abuse direction...

> 
> def on_key_up(key):
>  ??? global direction
> 
>  ??? #print('stop')
>  ??? direction = 'stop'
>  ??? stop_flag = True

In other words should forward_flag be set to false here?

> def update():
>  ??? global direction, stop_flag, forward_flag
> 
>  ??? if direction == 'stop' and stop_flag:
>  ??????? print('stop')
>  ??????? stop_flag = False
> 
>  ??? elif direction == 'forward' and forward_flag:
>  ??????? print('forward')
>  ??????? forward_flag = False
>  ??????? stop_flag = True
> 
>  ??? elif direction == 'quit':
>  ??????? quit()

This feels wrong to me. I'd expect it to look more like:

def update()

    if direction == forward and not forward_flag
       print direction
       forward_flag = True  # only set flag after first print
    elif direction == stop and not stop_flag
       print direction
       stop_flag = true   # same again
    elif direction == quit     # do you really want this in update?
       quit


I hope these random musings are helpful...
My main suggestions are:

Take the print and setting of the flags out of the handlers
and put that in update. Only set direction in the handlers.
And also make sure you set all of the state flags to their
correct values when you change state.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avi.e.gross at gmail.com  Sat Jun 18 01:22:57 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Sat, 18 Jun 2022 01:22:57 -0400
Subject: [Tutor] Event loop logic question
In-Reply-To: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
Message-ID: <050f01d882d3$789af080$69d0d180$@gmail.com>

Phil,

I am not familiar with the package you are using and have to assume you have some code somewhere not shown that binds your functions to the task as event handlers. 

But can multiple functions be called and active at the same time. I do not see any kind of lock in place such as a semaphore that stops two or more changes to the global variables as might happen if an arrow key is held down or whatever.

And your on_key_up() function does not make stop_flag global and so any changes it makes are to a local variable that immediately disappears. 

It is not my place to mention that this may be an advanced question and someone may want to direct you to resources showing the care that must be taken when a global variable is being handled by multiple threads. Ditto for forward_flag in on_key_down()

-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of Phil
Sent: Saturday, June 18, 2022 1:05 AM
To: tutor at python.org
Subject: [Tutor] Event loop logic question

Thank you for reading this.

What I'm trying to achieve here is have the word 'stop' printed once whenever no keys are pressed and 'forward' printed once even when the up arrow key is held down.

I don't think this question needs any knowledge of Pygame Zero to solve and it's quite likely that I've  misunderstood the logic required. The following code is my most recent version which seems to work the first time through the loop but fails after subsequent 'up' key presses. 
Debugging the event loop is difficult.

Any hints or alternative methods will be, as always, greatly appreciated.

By the way, this project sends commands to a controller which is easily overwhelmed if too many commands are received per second.

import pgzrun


WIDTH = 20
HEIGHT = 20

direction = 'stop'
stop_flag = True
forward_flag = True

def on_key_down(key):
     global direction

     if key == keys.UP:
         print('up pressed')
         direction = 'forward'
         forward_flag = True
         print('forward_flag ', forward_flag)

     elif key == keys.SPACE:
         direction = 'quit'

def on_key_up(key):
     global direction

     #print('stop')
     direction = 'stop'
     stop_flag = True

def update():
     global direction, stop_flag, forward_flag

     if direction == 'stop' and stop_flag:
         print('stop')
         stop_flag = False

     elif direction == 'forward' and forward_flag:
         print('forward')
         forward_flag = False
         stop_flag = True

     elif direction == 'quit':
         quit()

pgzrun.go()

--
Regards,
Phil

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From wlfraed at ix.netcom.com  Sat Jun 18 15:19:39 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Sat, 18 Jun 2022 15:19:39 -0400
Subject: [Tutor] Event loop logic question
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
Message-ID: <ls7sahpm4ro412kbap6b7ee6euamur51o7@4ax.com>

On Sat, 18 Jun 2022 15:04:50 +1000, Phil <phillor9 at gmail.com> declaimed the
following:


>Any hints or alternative methods will be, as always, greatly appreciated.
>
>By the way, this project sends commands to a controller which is easily 
>overwhelmed if too many commands are received per second.
>

	For the last matter, one solution would be to use an elapsed time
counter.

	rate = 1.0 / commands_per_second
	last_command_time = 0.0	#or get the time (fractional seconds) 
								#at start of program

	Then, when you issue a command, some logic like

	if last_command_time + rate < time_now:
		issue_command
		last_command_time = time_now

... optional if you return a status that "command skipped -- rate limited"


	For the former/general... I see too many flags that should (as I
understand it) be mutually exclusive (only one should be set/cleared
[depending on overall logic] at any moment in the program). Instead I'd
suggest using a "state" variable (maybe check on what Python has for
"enums" -- I've not used them in my dribblings, so the following is just
using "constants"

	STATE_QUIT, STATE_FORWARD, STATE_STOPPED = range(3)

	current_state = STATE_STOPPED
	requested_state = STATE_STOPPED

...	key-press
	if key == keys.UP:
		requested_state = STATE_FORWARD
	elif key == keys.SPACE:
		requested_state = STATE_QUIT
	else:
		pass	#log an unused key press		

...	key-release
	requested_state = STATE_STOPPED

...	update section

	if requested_state != current_state:
		#if the requested state is the same as current_state, 
		#it may signal a repeating key (but only if the OS doesn't
		#issue a key release and key press at the start of each repeat --
		#if it does -- you'll need to incorporate some nasty rate timer
		#to determine what is a repeat vs a real release/press)

		if requested_state == STATE_QUIT:
			#initiate shutdown?
			#log change?
		elif requested_state == STATE_FORWARD:
			#issue forward command to device
			#log change
		elif requested_state == STATE_STOPPED:
			#issue stop command to device
			#log change
		else:
			#unexpected STATE!
		current_state = requested_state	#update state



-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From tempest.amogh at gmail.com  Sun Jun 19 06:13:32 2022
From: tempest.amogh at gmail.com (tempest.amogh )
Date: Sun, 19 Jun 2022 15:43:32 +0530
Subject: [Tutor] Help with brute force algorithm
Message-ID: <78101919-6461-4009-b172-8996e3b23304@Canary>

As a personal project, I was inspired by Erik Demaine. He has created a font based on tetris shapes [http://erikdemaine.org/fonts/], however I would like to emulate this but based on another reference. I want to write an algorithm that brute force solves all the positions for these pieces in the english alphabet. I am having trouble making a grid that can be solved with these pieces, despite a book saying so.

I have attached the book that shows the shapes, and some pseudocode.

I would appreciate any possible guidance. Thank you very much :)

?
Regards,

Amogh Atwe

-------------- next part --------------
program start

import tetrominoes
import trominoes
import dominoes

input("Please write a word that you would like to see written").upper()

function(input):        
    for input{letter from alphabet}:
        
        backtracking algorithm, work from left to right:
            from filled letter to respective tetrominoes
            place(tetromino):
                for:
                    if tetromino does not fit in grid:
                        place in checked position:
                            place(tetromino)

                            repeat until pattern is found for letter, continue with rest of the letters
        
        print(letter from alphabet) in respective position with tetrominoes, trominoes and dominoes
        until no more letters


output (GUI of written word in combination of tetrominoes, trominoes, dominoes)

From manpritsinghece at gmail.com  Sun Jun 19 13:11:43 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Sun, 19 Jun 2022 22:41:43 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
Message-ID: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>

Dear Sir,

I was trying to make a sqlite3 data base from an ordinary text file named
abcd.txt on my computer. it has data as given below in the form of 2
columns

Time  Pieces
1       10
2       15
3       25
4       31
5       40
6       45
7       53
8       54
9       65
10      75

Below given is the code that makes  sqlite3 database work.db, with a table
named
workdata, which will be having 2 columns Time & Pieces . The code lateron
fetches all data from the table.

import sqlite3

def data_generator(workfile):
    with open(workfile) as obj:
        obj.readline()
        for line in obj:
            yield tuple(line.split())


con = sqlite3.connect("work.db")
cur = con.cursor()
cur.execute("create table workdata(Time INT, Pieces INT)")
cur.executemany("insert into workdata values (?, ?)",
data_generator("abcd.txt"))
con.commit()
cur.execute("select Time, Pieces from workdata")
print(cur.fetchall())
cur.close()
con.close()

In this example i am using generator function for iterator to be used in
executemany(). I will not prefer a list comprehension in this case as a
file may contain a large number of rows .

Is there any better way to do this task ? Kindly suggest.

Regards
Manprit Singh

From manpritsinghece at gmail.com  Sun Jun 19 13:52:05 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Sun, 19 Jun 2022 23:22:05 +0530
Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
Message-ID: <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>

Dear sir ,


One more thing :
def data_generator(workfile):
    with open(workfile) as obj:
        obj.readline()
        for line in obj:
            yield tuple(line.split())

Instead of yield tuple(line.split())    i can write yield line.split()
too.

Am i right ?

---------- Forwarded message ---------
From: Manprit Singh <manpritsinghece at gmail.com>
Date: Sun, Jun 19, 2022 at 10:41 PM
Subject: make a sqlite3 database from an ordinary text file
To: <tutor at python.org>


Dear Sir,

I was trying to make a sqlite3 data base from an ordinary text file named
abcd.txt on my computer. it has data as given below in the form of 2
columns

Time  Pieces
1       10
2       15
3       25
4       31
5       40
6       45
7       53
8       54
9       65
10      75

Below given is the code that makes  sqlite3 database work.db, with a table
named
workdata, which will be having 2 columns Time & Pieces . The code lateron
fetches all data from the table.

import sqlite3

def data_generator(workfile):
    with open(workfile) as obj:
        obj.readline()
        for line in obj:
            yield tuple(line.split())


con = sqlite3.connect("work.db")
cur = con.cursor()
cur.execute("create table workdata(Time INT, Pieces INT)")
cur.executemany("insert into workdata values (?, ?)",
data_generator("abcd.txt"))
con.commit()
cur.execute("select Time, Pieces from workdata")
print(cur.fetchall())
cur.close()
con.close()

In this example i am using generator function for iterator to be used in
executemany(). I will not prefer a list comprehension in this case as a
file may contain a large number of rows .

Is there any better way to do this task ? Kindly suggest.

Regards
Manprit Singh

From alan.gauld at yahoo.co.uk  Sun Jun 19 14:04:12 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 19 Jun 2022 19:04:12 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
Message-ID: <t8noes$103i$1@ciao.gmane.io>

On 19/06/2022 18:11, Manprit Singh wrote:

> cur.execute("create table workdata(Time INT, Pieces INT)")

This will fail the second time it runs. You should check
if the table already exists and either drop it or use
the existing table.

However, I rarely create tables from Python, it's generally
easier to create a sql file and run that directly using
the sqlite interpreter. You only need Python if the format
of the table has to be deduced from the data. and that then
leads to all sorts of follow-on issues over accessing names
of columns etc.

Once you have the table you can then use Python to load
the data, although in your case you could do that directly
from SQL too since it looks like a tab separated file.

As always use the most appropriate tool. If you can do it
directly in SQL there is no point in wrapping it in
Python. That just adds complexity and slows things
down.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Sun Jun 19 14:09:02 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 19 Jun 2022 19:09:02 +0100
Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>
Message-ID: <t8nonu$103i$3@ciao.gmane.io>

On 19/06/2022 18:52, Manprit Singh wrote:

> One more thing :
> def data_generator(workfile):
>     with open(workfile) as obj:
>         obj.readline()
>         for line in obj:
>             yield tuple(line.split())
> 
> Instead of yield tuple(line.split())    i can write yield line.split()
> too.
> 
> Am i right ?

It depends what you want. The first one gives a tuple
back, the second gives a list.

Often it won't make any difference but if you were using
the value as a dictionary key, say, then only the first
would work.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Sun Jun 19 14:07:01 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 19 Jun 2022 19:07:01 +0100
Subject: [Tutor] Help with brute force algorithm
In-Reply-To: <78101919-6461-4009-b172-8996e3b23304@Canary>
References: <78101919-6461-4009-b172-8996e3b23304@Canary>
Message-ID: <t8nok5$103i$2@ciao.gmane.io>

On 19/06/2022 11:13, tempest.amogh wrote:
> As a personal project, I was inspired by Erik Demaine. > He has created a font based on tetris shapes
>
> I am having trouble making a grid that can be solved 
> with these pieces, despite a book saying so.

So show us the code and sample data and any error messages.

> 
> I have attached the book that shows the shapes, and some pseudocode.

Non-text attachments get stripped by the server for security reasons.
Copy/paste the relevant text into the email or provide a link to a paste
bin or other web site.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From manpritsinghece at gmail.com  Sun Jun 19 14:33:27 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 00:03:27 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <t8noes$103i$1@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
Message-ID: <CAO1OCwYwr3LbpKKq6Hzt80w8cnZGEoyohBL75H8poALnN1yuMg@mail.gmail.com>

Yes sir ,
I do agree it will fail for second time , if table exists .

One treatment could be writing the command like create table table_name if
not exist (--------)

I do agree with your further comments too .

Regards
Manprit Singh

On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, <tutor at python.org> wrote:

> On 19/06/2022 18:11, Manprit Singh wrote:
>
> > cur.execute("create table workdata(Time INT, Pieces INT)")
>
> This will fail the second time it runs. You should check
> if the table already exists and either drop it or use
> the existing table.
>
> However, I rarely create tables from Python, it's generally
> easier to create a sql file and run that directly using
> the sqlite interpreter. You only need Python if the format
> of the table has to be deduced from the data. and that then
> leads to all sorts of follow-on issues over accessing names
> of columns etc.
>
> Once you have the table you can then use Python to load
> the data, although in your case you could do that directly
> from SQL too since it looks like a tab separated file.
>
> As always use the most appropriate tool. If you can do it
> directly in SQL there is no point in wrapping it in
> Python. That just adds complexity and slows things
> down.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From manpritsinghece at gmail.com  Sun Jun 19 14:37:42 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 00:07:42 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <t8noes$103i$1@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
Message-ID: <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>

Dear sir ,

One more query ....
How can i use column name in text file as colum names in sqlite3 database .
I tried using python but failed.

Can you put some light over it ?
Regards
Manprit Singh

On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, <tutor at python.org> wrote:

> On 19/06/2022 18:11, Manprit Singh wrote:
>
> > cur.execute("create table workdata(Time INT, Pieces INT)")
>
> This will fail the second time it runs. You should check
> if the table already exists and either drop it or use
> the existing table.
>
> However, I rarely create tables from Python, it's generally
> easier to create a sql file and run that directly using
> the sqlite interpreter. You only need Python if the format
> of the table has to be deduced from the data. and that then
> leads to all sorts of follow-on issues over accessing names
> of columns etc.
>
> Once you have the table you can then use Python to load
> the data, although in your case you could do that directly
> from SQL too since it looks like a tab separated file.
>
> As always use the most appropriate tool. If you can do it
> directly in SQL there is no point in wrapping it in
> Python. That just adds complexity and slows things
> down.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From alan.gauld at yahoo.co.uk  Sun Jun 19 15:16:39 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 19 Jun 2022 20:16:39 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
Message-ID: <t8nsmn$4t3$1@ciao.gmane.io>

On 19/06/2022 19:37, Manprit Singh wrote:
> How can i use column name in text file as colum names in sqlite3 database .
> I tried using python but failed.

I'm not sure if this is what you mean but....


sqlite> .help import
.import FILE TABLE       Import data from FILE into TABLE
   Options:
     --ascii               Use \037 and \036 as column and row separators
     --csv                 Use , and \n as column and row separators
     --skip N              Skip the first N rows of input
     -v                    "Verbose" - increase auxiliary output
   Notes:
     *  If TABLE does not exist, it is created.  The first row of input
        determines the column names.
     *  If neither --csv or --ascii are used, the input mode is derived
        from the ".mode" output mode
     *  If FILE begins with "|" then it is a command that generates the
        input text.
sqlite>



-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From wlfraed at ix.netcom.com  Sun Jun 19 15:17:06 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Sun, 19 Jun 2022 15:17:06 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwYwr3LbpKKq6Hzt80w8cnZGEoyohBL75H8poALnN1yuMg@mail.gmail.com>
Message-ID: <iftuahd43v682dmml5tos52gqdtil8qefl@4ax.com>

On Mon, 20 Jun 2022 00:03:27 +0530, Manprit Singh
<manpritsinghece at gmail.com> declaimed the following:

>Yes sir ,
>I do agree it will fail for second time , if table exists .
>
>One treatment could be writing the command like create table table_name if
>not exist (--------)
>

	The syntax for that is

		CREATE TABLE IF NOT EXISTS <table-name> ...


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From wlfraed at ix.netcom.com  Sun Jun 19 15:26:46 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Sun, 19 Jun 2022 15:26:46 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
Message-ID: <7ituahdsjskfm8pv9dn80sph2kiqrtevv5@4ax.com>

On Mon, 20 Jun 2022 00:07:42 +0530, Manprit Singh
<manpritsinghece at gmail.com> declaimed the following:

>Dear sir ,
>
>One more query ....
>How can i use column name in text file as colum names in sqlite3 database .
>I tried using python but failed.

	Show us the code, and the traceback... Otherwise we have no real idea
of what you might be doing...

	Not to mention -- you are opening yourself up for an SQL injection
attack by doing what I think you mean.

	ANYTHING involving the database schema should NEVER rely upon user
input values. Only the DATA, and that via the adapter's form of
parameterization (SQLite3 using ?, others have different syntax). This way
the adapter can do escaping and quoting if needed to ensure that the value
is treated solely as data, and not something that could be interpreted as
another SQL statement.

	If you need to generate SQL statements from user specified fields, you
should present a menu to the user of the fields that are defined in the
database, and use the user input (position/index #) to select from the
known field names.


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From avi.e.gross at gmail.com  Sun Jun 19 19:43:14 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Sun, 19 Jun 2022 19:43:14 -0400
Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file
In-Reply-To: <t8nonu$103i$3@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>
 <t8nonu$103i$3@ciao.gmane.io>
Message-ID: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com>

I may not be looking at this right since much happens in functions not seen
but does cur.execute(many) expect two items of text or two items of type
integer or will it handle a list or tuple with arbitrary numbers of contents
and so on, is what makes the design decision.

Deeper within that function, it presumable maps your passed function to code
that may do,

First, Second = f()

And so on. Or it may just take a single result or other choices. It is a bit
confusing to pass a filename as someone pointed out. 

In what follows, either I am confused or the one asking the question is.

The goal of the iterator seems to be to avoid reading all the data in at
once, right. Otherwise, why bother when you can do something like reading in
the entire file and splitting and putting the results in something like a
list structure, or perhaps in something from numpy.

So why does the code use readline() (no 's') to read in a single line and
only once? 

As I read the somewhat confusing code, the goal is to open the file the
first time and read a line and throw the results away as it is not clear
where it is saved. Then the goal is to loop on nothing (or maybe one line)
and yield the results of splitting it and pause till called again. But since
only no or maybe one line have been received, the call to iterate should
just exit as the loop is done.

So I think reading a single line should be moved down within the loop!

And you need some kind of test to know when there are no remaining lines in
the file or the current line is empty and instead of yielding a non-existent
result, you should return what the iteration protocol needs and close the
file, albeit the with does that for you silently as part of that protocol.

But as others have noted, it is not clear other than as an exercise, if this
iterator is needed. Your other modules may supply something for you.

Where a generator like this is even more useful is when at some point you
want to stop and perhaps resume a bit later elsewhere. For example, if your
code wanted random numbers that had been precalculated and use it for
various purposes until done, the iterator can in principle be called from
multiple places and keep marching along and eventually stop without reading
the entire file. Another example would be a generator that yields all primes
needed in order, and generally that never goes long enough to run out of
primes, LOL!




-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Alan Gauld via Tutor
Sent: Sunday, June 19, 2022 2:09 PM
To: tutor at python.org
Subject: Re: [Tutor] Fwd: make a sqlite3 database from an ordinary text file

On 19/06/2022 18:52, Manprit Singh wrote:

> One more thing :
> def data_generator(workfile):
>     with open(workfile) as obj:
>         obj.readline()
>         for line in obj:
>             yield tuple(line.split())
> 
> Instead of yield tuple(line.split())    i can write yield line.split()
> too.
> 
> Am i right ?

It depends what you want. The first one gives a tuple back, the second gives
a list.

Often it won't make any difference but if you were using the value as a
dictionary key, say, then only the first would work.

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avi.e.gross at gmail.com  Sun Jun 19 19:57:51 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Sun, 19 Jun 2022 19:57:51 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
Message-ID: <007e01d88438$63156090$294021b0$@gmail.com>

Manprit,

You may be approaching this topic differently than you could.

What you keep calling a TEXT file is what others might call structured files
like .CSV or .DAT types that contain values separated by some delimiter like
a comma or a tab or arbitrary whitespace or other choices. Lots of python
functionalities are available to read  these in to structures used in
modules like numpy and pandas that many people use on top of or alongside
regular built-in python commands. In these, there can be (and usually are)
HEADER names for columns and sometimes also rows. The functionality that
reads in such tables, Dataframes, or whatever you want to call it, will
often optionally take the first line of the file to have headers specifying
names. And, you can always add or change the names as well.

So if passing the entire structure to another function, it can choose to
access the name of any column (or often row) when it wants. Or, if you need
to, you can ask for it and pass it your way as in interpolating the name
into an SQL command.

You may be asking a slightly different question here about just your data.
Does your "text" file contain a first line saying ""Time	Pieces" or
not?

If it DOES, you may need to read it ONCE at the start before giving your
iterator for the numbers to be read and store the results as your names to
use as you wish. If the file has no header, then your code is the one
attaching a name.

To me the real issue is your design seems to be of a limited nature. You
want to read in your data in a small batch (single line) and presumably are
opening a connection to the database and doing login and so on, then sending
a command to create a table, unless it already exists, then sending umpteen
commands to insert yet another row into the table and then who knows? This
seems like such a common operation that I have to guess there is a way to do
it fairly trivially and my guess is way more efficiently.



-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Manprit Singh
Sent: Sunday, June 19, 2022 2:38 PM
Cc: tutor at python.org
Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file

Dear sir ,

One more query ....
How can i use column name in text file as colum names in sqlite3 database .
I tried using python but failed.

Can you put some light over it ?
Regards
Manprit Singh

On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, <tutor at python.org> wrote:

> On 19/06/2022 18:11, Manprit Singh wrote:
>
> > cur.execute("create table workdata(Time INT, Pieces INT)")
>
> This will fail the second time it runs. You should check if the table 
> already exists and either drop it or use the existing table.
>
> However, I rarely create tables from Python, it's generally easier to 
> create a sql file and run that directly using the sqlite interpreter. 
> You only need Python if the format of the table has to be deduced from 
> the data. and that then leads to all sorts of follow-on issues over 
> accessing names of columns etc.
>
> Once you have the table you can then use Python to load the data, 
> although in your case you could do that directly from SQL too since it 
> looks like a tab separated file.
>
> As always use the most appropriate tool. If you can do it directly in 
> SQL there is no point in wrapping it in Python. That just adds 
> complexity and slows things down.
>
> --
> Alan G
> Author of the Learn to Program web site http://www.alan-g.me.uk/ 
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From manpritsinghece at gmail.com  Sun Jun 19 20:59:58 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 06:29:58 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <007e01d88438$63156090$294021b0$@gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
Message-ID: <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>

Dear all,
Learning is a lifelong process . At this point i still do not feel i am
absolutely ok with python or programming. In the past i was very active in
this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber,
dn and few others, they always answered my queries .
As i wrote in the starting , i have a simple text file named abcd.txt. (I
am not using a csv file or any other structured file) . The contents are
given below :

Time  Pieces
1       10
2       15
3       25
4       31
5       40
6       45
7       53
8       54
9       65
10      75

We can clearly see the first line of the file is

Time  Pieces

Now I have to read this text  into an sqlite3 database table , So it is
clear that the database table will have 2 columns, Column names of the
sqlite3 table are to be picked from the  first line of the text file
abcd.txt. (First column name will be Time, second column name will be
Pieces ).
See i know this can be done very easily using numpy and Pandas. But I am
trying to do it with Core Python only.

cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))

I just want to insert column names (where these column names are  read
from text file abcd.txt 1st line)  using a question mark style
placeholder.  Pls let me know if anything is possible .

Time  Pieces

On Mon, Jun 20, 2022 at 5:34 AM <avi.e.gross at gmail.com> wrote:

> Manprit,
>
> You may be approaching this topic differently than you could.
>
> What you keep calling a TEXT file is what others might call structured
> files
> like .CSV or .DAT types that contain values separated by some delimiter
> like
> a comma or a tab or arbitrary whitespace or other choices. Lots of python
> functionalities are available to read  these in to structures used in
> modules like numpy and pandas that many people use on top of or alongside
> regular built-in python commands. In these, there can be (and usually are)
> HEADER names for columns and sometimes also rows. The functionality that
> reads in such tables, Dataframes, or whatever you want to call it, will
> often optionally take the first line of the file to have headers specifying
> names. And, you can always add or change the names as well.
>
> So if passing the entire structure to another function, it can choose to
> access the name of any column (or often row) when it wants. Or, if you need
> to, you can ask for it and pass it your way as in interpolating the name
> into an SQL command.
>
> You may be asking a slightly different question here about just your data.
> Does your "text" file contain a first line saying ""Time        Pieces" or
> not?
>
> If it DOES, you may need to read it ONCE at the start before giving your
> iterator for the numbers to be read and store the results as your names to
> use as you wish. If the file has no header, then your code is the one
> attaching a name.
>
> To me the real issue is your design seems to be of a limited nature. You
> want to read in your data in a small batch (single line) and presumably are
> opening a connection to the database and doing login and so on, then
> sending
> a command to create a table, unless it already exists, then sending umpteen
> commands to insert yet another row into the table and then who knows? This
> seems like such a common operation that I have to guess there is a way to
> do
> it fairly trivially and my guess is way more efficiently.
>
>
>
> -----Original Message-----
> From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
> Manprit Singh
> Sent: Sunday, June 19, 2022 2:38 PM
> Cc: tutor at python.org
> Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file
>
> Dear sir ,
>
> One more query ....
> How can i use column name in text file as colum names in sqlite3 database .
> I tried using python but failed.
>
> Can you put some light over it ?
> Regards
> Manprit Singh
>
> On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, <tutor at python.org>
> wrote:
>
> > On 19/06/2022 18:11, Manprit Singh wrote:
> >
> > > cur.execute("create table workdata(Time INT, Pieces INT)")
> >
> > This will fail the second time it runs. You should check if the table
> > already exists and either drop it or use the existing table.
> >
> > However, I rarely create tables from Python, it's generally easier to
> > create a sql file and run that directly using the sqlite interpreter.
> > You only need Python if the format of the table has to be deduced from
> > the data. and that then leads to all sorts of follow-on issues over
> > accessing names of columns etc.
> >
> > Once you have the table you can then use Python to load the data,
> > although in your case you could do that directly from SQL too since it
> > looks like a tab separated file.
> >
> > As always use the most appropriate tool. If you can do it directly in
> > SQL there is no point in wrapping it in Python. That just adds
> > complexity and slows things down.
> >
> > --
> > Alan G
> > Author of the Learn to Program web site http://www.alan-g.me.uk/
> > http://www.amazon.com/author/alan_gauld
> > Follow my photo-blog on Flickr at:
> > http://www.flickr.com/photos/alangauldphotos
> >
> >
> > _______________________________________________
> > Tutor maillist  -  Tutor at python.org
> > To unsubscribe or change subscription options:
> > https://mail.python.org/mailman/listinfo/tutor
> >
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From PythonList at DancesWithMice.info  Sun Jun 19 22:39:14 2022
From: PythonList at DancesWithMice.info (dn)
Date: Mon, 20 Jun 2022 14:39:14 +1200
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
Message-ID: <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>

On 20/06/2022 12.59, Manprit Singh wrote:
> Dear all,
> Learning is a lifelong process . At this point i still do not feel i am
> absolutely ok with python or programming. In the past i was very active in
> this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber,
> dn and few others, they always answered my queries .
> As i wrote in the starting , i have a simple text file named abcd.txt. (I
> am not using a csv file or any other structured file) . The contents are
> given below :
> 
> Time  Pieces
> 1       10
> 2       15
> 3       25
> 4       31
> 5       40
> 6       45
> 7       53
> 8       54
> 9       65
> 10      75
> 
> We can clearly see the first line of the file is
> 
> Time  Pieces

but, as advised earlier, whilst "we can clearly see" is true IN THIS
CASE; what happens when someone brings another table to the party?

Ref: https://xkcd.com/327/

BTW my favorite joke from this genre is:

    ? = 3.141592653589793helpimtrappedinapiefactory7108914...


I'm +1 after @Alan. Although we often see it in programming texts, I've
NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS
and SQL very early in my career, and therefore know to use the tools
from that eco-system. (the rationale in books is the exact opposite -
presuming the reader has never used DB-tools). Use the best tool for the
job!


With regard to "Learning is a lifelong process", most?all of us here
will whole-heartedly agree. Those who don't have little reason to
subscribe to this list!

However, rather than spending years (it must surely be) working on small
and simple (and at-times, artificial or academic) problems - even to the
point of seeking the most efficient of solutions (to same); there comes
a time to stop "learning" per-se, and start 'doing'!

The best way to learn how to USE Python to solve 'real-world' problems
is to use it - in 'the real-world'! Pick a project that will require
multiple classes, several modules, or hundreds of lines of code; and
dive in!

Given that we've ventured into a multi-media discussion. Here is the
very first PyBites podcast (out of more than 70, over the years). It
espouses a solid 'learn by doing' mind-set and (IIRC) talks of
"JiT-learning" (start from a solid base, and then learn what you
need-to, when you need to): https://www.youtube.com/watch?v=NoTkCSP3A68
- or in a component of my understanding of "refactoring": 'there must be
a better way'...

-- 
Regards,
=dn

From alan.gauld at yahoo.co.uk  Mon Jun 20 06:54:03 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 20 Jun 2022 11:54:03 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
Message-ID: <t8pjkb$ut9$1@ciao.gmane.io>

On 20/06/2022 01:59, Manprit Singh wrote:

> As i wrote in the starting , i have a simple text file named abcd.txt. (I
> am not using a csv file or any other structured file) . The contents are
> given below :
> 
> Time  Pieces
> 1       10
> 2       15
> 3       25
> 4       31
> 5       40
> 6       45
> 7       53
> 8       54
> 9       65
> 10      75

Sticking with the theme of using the right tool for the right
job here is how I did this on Unix(after cutting and pasting
your daata into a file called data.txt):

~/src/sql $ tr -cs "[:alnum:]\n" "," <data.txt >data2.txt
~/src/sql $ sqlite3

sqlite> .import -csv data2.txt DATA

sqlite> .schema DATA
CREATE TABLE IF NOT EXISTS "DATA"(
  "Time" TEXT,
  "Pieces" TEXT
);

sqlite> Select Time from DATA where Pieces=40;
5

ie.
I used tr to convert the text file to a CSV.
(-cs means translate everything except alphanum and newlines
and "squeeze" to one replacement character)


Then I imported the csv file into SQLite.

To test it, I checked the schema create statement
and did a select query.

If you don't like tr you could of course use a simple python
(or perl or awk or sed) script to replace the spaces with , or
even use a spreadsheet such as Excel. In Python:

>>> with open('data.txt') as inf:
         with open('data2.txt",'w') as outf:
            for line in inf:
                line = re.sub(" +", ",", line)
                outf.write(line)


The only problem with this approach is that it makes the
two columns TEXT rather than INT.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Mon Jun 20 07:07:48 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 20 Jun 2022 12:07:48 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
Message-ID: <t8pke5$jp3$1@ciao.gmane.io>

On 20/06/2022 01:59, Manprit Singh wrote:

> clear that the database table will have 2 columns, Column names of the
> sqlite3 table are to be picked from the  first line of the text file
> abcd.txt. (First column name will be Time, second column name will be
> Pieces ).
> See i know this can be done very easily using numpy and Pandas. But I am
> trying to do it with Core Python only.
> 
> cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
> 
> I just want to insert column names (where these column names are  read
> from text file abcd.txt 1st line)  using a question mark style
> placeholder. 

OK, Having shown how I would really do it, I'll now try to
answer the question for pure Python:

def get_data(open_file):
    for line in open_file:
        yield line.split()


# open the database/cursor here....

with open('data.txt') as inf:
    heads = inf.readline().split()
    cur.execute("create table DATA(? INT, ? INT)",heads)
    cur.executemany("insert into workdata values (?, ?)",get_data(inf))

cur.execute("select * from DATA")
for row in cur.fetchall():
    print(row)


Note, I didn't actually run this but I think it should be close.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From manpritsinghece at gmail.com  Mon Jun 20 10:17:51 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 19:47:51 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <t8pke5$jp3$1@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <t8pke5$jp3$1@ciao.gmane.io>
Message-ID: <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>

Dear Sir,

Many thanks for this . This mail has a lot of things for me . I Will do all
the exercises on my own and will revert you soon.

Regards
Manprit Singh

On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor <tutor at python.org>
wrote:

> On 20/06/2022 01:59, Manprit Singh wrote:
>
> > clear that the database table will have 2 columns, Column names of the
> > sqlite3 table are to be picked from the  first line of the text file
> > abcd.txt. (First column name will be Time, second column name will be
> > Pieces ).
> > See i know this can be done very easily using numpy and Pandas. But I am
> > trying to do it with Core Python only.
> >
> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
> >
> > I just want to insert column names (where these column names are  read
> > from text file abcd.txt 1st line)  using a question mark style
> > placeholder.
>
> OK, Having shown how I would really do it, I'll now try to
> answer the question for pure Python:
>
> def get_data(open_file):
>     for line in open_file:
>         yield line.split()
>
>
> # open the database/cursor here....
>
> with open('data.txt') as inf:
>     heads = inf.readline().split()
>     cur.execute("create table DATA(? INT, ? INT)",heads)
>     cur.executemany("insert into workdata values (?, ?)",get_data(inf))
>
> cur.execute("select * from DATA")
> for row in cur.fetchall():
>     print(row)
>
>
> Note, I didn't actually run this but I think it should be close.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From manpritsinghece at gmail.com  Mon Jun 20 10:34:52 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 20:04:52 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <t8pke5$jp3$1@ciao.gmane.io>
 <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>
Message-ID: <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>

 Dear Sir ,



 cur.execute("create table DATA(? INT, ? INT)",heads)

As in earlier mail I pointed out that the above command  will not work
because column names are not written inside the  single quotes in the
command create table.  Hence we can not use (?} place holders  .

Regards
Manprit Singh

On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh <manpritsinghece at gmail.com>
wrote:

> Dear Sir,
>
> Many thanks for this . This mail has a lot of things for me . I Will do
> all the exercises on my own and will revert you soon.
>
> Regards
> Manprit Singh
>
> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor <tutor at python.org>
> wrote:
>
>> On 20/06/2022 01:59, Manprit Singh wrote:
>>
>> > clear that the database table will have 2 columns, Column names of the
>> > sqlite3 table are to be picked from the  first line of the text file
>> > abcd.txt. (First column name will be Time, second column name will be
>> > Pieces ).
>> > See i know this can be done very easily using numpy and Pandas. But I am
>> > trying to do it with Core Python only.
>> >
>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
>> >
>> > I just want to insert column names (where these column names are  read
>> > from text file abcd.txt 1st line)  using a question mark style
>> > placeholder.
>>
>> OK, Having shown how I would really do it, I'll now try to
>> answer the question for pure Python:
>>
>> def get_data(open_file):
>>     for line in open_file:
>>         yield line.split()
>>
>>
>> # open the database/cursor here....
>>
>> with open('data.txt') as inf:
>>     heads = inf.readline().split()
>>     cur.execute("create table DATA(? INT, ? INT)",heads)
>>     cur.executemany("insert into workdata values (?, ?)",get_data(inf))
>>
>> cur.execute("select * from DATA")
>> for row in cur.fetchall():
>>     print(row)
>>
>>
>> Note, I didn't actually run this but I think it should be close.
>>
>> --
>> Alan G
>> Author of the Learn to Program web site
>> http://www.alan-g.me.uk/
>> http://www.amazon.com/author/alan_gauld
>> Follow my photo-blog on Flickr at:
>> http://www.flickr.com/photos/alangauldphotos
>>
>>
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>

From manpritsinghece at gmail.com  Mon Jun 20 11:32:56 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 21:02:56 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <t8pke5$jp3$1@ciao.gmane.io>
 <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>
 <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>
Message-ID: <CAO1OCwYv4iN3kzQ+Wm5HG53SBnPknu3-5pn8D6TOgLAbK380Ww@mail.gmail.com>

Dear Sir,

The most convenient way that i found to read that text file into an sqlite3
database is using pandas as given below:

import sqlite3
import pandas as pd

df = pd.read_table("abcd.txt", sep="\s+")
                                      # will read the entire file to a
dataframe
conn = sqlite3.connect("work.db")
                                         #  will make a database  work.db
cur = conn.cursor()
df.to_sql(name="worktime", con=conn, if_exists="replace",
index=False)                      # Entire text file data, which is into a
dataframe is now written to sqlite table
cur.execute("select Time from worktime where
Pieces=40")                                          # select query
print(cur.fetchall())

gives output :

[(5,)]


On Mon, Jun 20, 2022 at 8:04 PM Manprit Singh <manpritsinghece at gmail.com>
wrote:

>  Dear Sir ,
>
>
>
>  cur.execute("create table DATA(? INT, ? INT)",heads)
>
> As in earlier mail I pointed out that the above command  will not work
> because column names are not written inside the  single quotes in the
> command create table.  Hence we can not use (?} place holders  .
>
> Regards
> Manprit Singh
>
> On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh <manpritsinghece at gmail.com>
> wrote:
>
>> Dear Sir,
>>
>> Many thanks for this . This mail has a lot of things for me . I Will do
>> all the exercises on my own and will revert you soon.
>>
>> Regards
>> Manprit Singh
>>
>> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor <tutor at python.org>
>> wrote:
>>
>>> On 20/06/2022 01:59, Manprit Singh wrote:
>>>
>>> > clear that the database table will have 2 columns, Column names of the
>>> > sqlite3 table are to be picked from the  first line of the text file
>>> > abcd.txt. (First column name will be Time, second column name will be
>>> > Pieces ).
>>> > See i know this can be done very easily using numpy and Pandas. But I
>>> am
>>> > trying to do it with Core Python only.
>>> >
>>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
>>> >
>>> > I just want to insert column names (where these column names are  read
>>> > from text file abcd.txt 1st line)  using a question mark style
>>> > placeholder.
>>>
>>> OK, Having shown how I would really do it, I'll now try to
>>> answer the question for pure Python:
>>>
>>> def get_data(open_file):
>>>     for line in open_file:
>>>         yield line.split()
>>>
>>>
>>> # open the database/cursor here....
>>>
>>> with open('data.txt') as inf:
>>>     heads = inf.readline().split()
>>>     cur.execute("create table DATA(? INT, ? INT)",heads)
>>>     cur.executemany("insert into workdata values (?, ?)",get_data(inf))
>>>
>>> cur.execute("select * from DATA")
>>> for row in cur.fetchall():
>>>     print(row)
>>>
>>>
>>> Note, I didn't actually run this but I think it should be close.
>>>
>>> --
>>> Alan G
>>> Author of the Learn to Program web site
>>> http://www.alan-g.me.uk/
>>> http://www.amazon.com/author/alan_gauld
>>> Follow my photo-blog on Flickr at:
>>> http://www.flickr.com/photos/alangauldphotos
>>>
>>>
>>> _______________________________________________
>>> Tutor maillist  -  Tutor at python.org
>>> To unsubscribe or change subscription options:
>>> https://mail.python.org/mailman/listinfo/tutor
>>>
>>

From manpritsinghece at gmail.com  Mon Jun 20 11:41:51 2022
From: manpritsinghece at gmail.com (Manprit Singh)
Date: Mon, 20 Jun 2022 21:11:51 +0530
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwYv4iN3kzQ+Wm5HG53SBnPknu3-5pn8D6TOgLAbK380Ww@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <t8pke5$jp3$1@ciao.gmane.io>
 <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>
 <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>
 <CAO1OCwYv4iN3kzQ+Wm5HG53SBnPknu3-5pn8D6TOgLAbK380Ww@mail.gmail.com>
Message-ID: <CAO1OCwYK2inbBaso0QGNbU4J05KbGtVcrAGY84+WbkiG6j8BLg@mail.gmail.com>

Dear Sir,

Also the column names in the text file, becomes the column name in sqlite3
table. I have verified it while opening the table in column mode from
sqlite3 terminal

On Mon, Jun 20, 2022 at 9:02 PM Manprit Singh <manpritsinghece at gmail.com>
wrote:

> Dear Sir,
>
> The most convenient way that i found to read that text file into an
> sqlite3 database is using pandas as given below:
>
> import sqlite3
> import pandas as pd
>
> df = pd.read_table("abcd.txt", sep="\s+")
>                                       # will read the entire file to a
> dataframe
> conn = sqlite3.connect("work.db")
>                                          #  will make a database  work.db
> cur = conn.cursor()
> df.to_sql(name="worktime", con=conn, if_exists="replace",
> index=False)                      # Entire text file data, which is into a
> dataframe is now written to sqlite table
> cur.execute("select Time from worktime where
> Pieces=40")                                          # select query
> print(cur.fetchall())
>
> gives output :
>
> [(5,)]
>
>
> On Mon, Jun 20, 2022 at 8:04 PM Manprit Singh <manpritsinghece at gmail.com>
> wrote:
>
>>  Dear Sir ,
>>
>>
>>
>>  cur.execute("create table DATA(? INT, ? INT)",heads)
>>
>> As in earlier mail I pointed out that the above command  will not work
>> because column names are not written inside the  single quotes in the
>> command create table.  Hence we can not use (?} place holders  .
>>
>> Regards
>> Manprit Singh
>>
>> On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh <manpritsinghece at gmail.com>
>> wrote:
>>
>>> Dear Sir,
>>>
>>> Many thanks for this . This mail has a lot of things for me . I Will do
>>> all the exercises on my own and will revert you soon.
>>>
>>> Regards
>>> Manprit Singh
>>>
>>> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor <tutor at python.org>
>>> wrote:
>>>
>>>> On 20/06/2022 01:59, Manprit Singh wrote:
>>>>
>>>> > clear that the database table will have 2 columns, Column names of the
>>>> > sqlite3 table are to be picked from the  first line of the text file
>>>> > abcd.txt. (First column name will be Time, second column name will be
>>>> > Pieces ).
>>>> > See i know this can be done very easily using numpy and Pandas. But I
>>>> am
>>>> > trying to do it with Core Python only.
>>>> >
>>>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
>>>> >
>>>> > I just want to insert column names (where these column names are  read
>>>> > from text file abcd.txt 1st line)  using a question mark style
>>>> > placeholder.
>>>>
>>>> OK, Having shown how I would really do it, I'll now try to
>>>> answer the question for pure Python:
>>>>
>>>> def get_data(open_file):
>>>>     for line in open_file:
>>>>         yield line.split()
>>>>
>>>>
>>>> # open the database/cursor here....
>>>>
>>>> with open('data.txt') as inf:
>>>>     heads = inf.readline().split()
>>>>     cur.execute("create table DATA(? INT, ? INT)",heads)
>>>>     cur.executemany("insert into workdata values (?, ?)",get_data(inf))
>>>>
>>>> cur.execute("select * from DATA")
>>>> for row in cur.fetchall():
>>>>     print(row)
>>>>
>>>>
>>>> Note, I didn't actually run this but I think it should be close.
>>>>
>>>> --
>>>> Alan G
>>>> Author of the Learn to Program web site
>>>> http://www.alan-g.me.uk/
>>>> http://www.amazon.com/author/alan_gauld
>>>> Follow my photo-blog on Flickr at:
>>>> http://www.flickr.com/photos/alangauldphotos
>>>>
>>>>
>>>> _______________________________________________
>>>> Tutor maillist  -  Tutor at python.org
>>>> To unsubscribe or change subscription options:
>>>> https://mail.python.org/mailman/listinfo/tutor
>>>>
>>>

From wlfraed at ix.netcom.com  Mon Jun 20 12:18:08 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Mon, 20 Jun 2022 12:18:08 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
Message-ID: <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com>

On Mon, 20 Jun 2022 06:29:58 +0530, Manprit Singh
<manpritsinghece at gmail.com> declaimed the following:

>See i know this can be done very easily using numpy and Pandas. But I am
>trying to do it with Core Python only.
>
>cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
>
>I just want to insert column names (where these column names are  read
>from text file abcd.txt 1st line)  using a question mark style
>placeholder.  Pls let me know if anything is possible .
>

	You DON'T!

	As stated in my previous post, that is manipulating the database schema
-- which should never rely upon values found in data files due to the
possibility of an SQL injection attack.

https://www.explainxkcd.com/wiki/index.php/327:_Exploits_of_a_Mom

	The purpose of using the parameterized placeholders is that the
interface can "sanitize" (escape/quote) characters in the DATA that have
meaning in SQL.

	The SQLite3 adapter appears to inherently block using parameterized
queries to modify the database schema. If you REALLY must generate the
schema using data from a file, you have to do all of the that in Python,
BEFORE passing it to .execute().

>>> import sqlite3 as db
>>> con = db.connect("junk.db")
>>> cur = con.cursor()
>>> cur.execute("create table workdata (? INT, ? INT)", ("Time", "Pieces"))
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
sqlite3.OperationalError: near "?": syntax error
>>> cur.execute("create table workdata (%s INT, %s INT)" % ("Time", "Pieces"))
<sqlite3.Cursor object at 0x000002B642A8D030>
>>> cur.execute("insert into workdata (Time, Pieces) values (?, ?)", (123, "who"))
<sqlite3.Cursor object at 0x000002B642A8D030>
>>>

	Note that the second .execute() is using Python string interpolation to
create the schema definition SQL, not parameterized SQLite3 operations. The
third .execute() shows that parameterized statements work with DATA.  And,
per the warning about SQL injection attacks, you should NEVER use string
interpolation (or other formatting operations) in Python to work with data
for the database.


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From alan.gauld at yahoo.co.uk  Mon Jun 20 13:32:20 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 20 Jun 2022 18:32:20 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com>
Message-ID: <t8qav4$pa8$1@ciao.gmane.io>

On 20/06/2022 17:18, Dennis Lee Bieber wrote:

> 	The SQLite3 adapter appears to inherently block using parameterized
> queries to modify the database schema. If you REALLY must generate the
> schema using data from a file, you have to do all of the that in Python,

Ah! I didn't know that. Precisely because I never use Python
to create a database!

> per the warning about SQL injection attacks, you should NEVER use string
> interpolation (or other formatting operations) in Python to work with data
> for the database.

That's true, although I might not say NEVER, since often you write
one-off scripts for database work that nobody else ever uses. But the
official mechanism works well enough for most real cases that there's
rarely any reason not to use it.

And creating tables and columns dynamically is frought with problems
too. It's like trying to create variables in code from user input - how
does the rest of the code refer to those values? For variables we can
use a dictionary as a partial solution but for a database it gets very
messy with the need to look at the meta data before performing any action.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avi.e.gross at gmail.com  Sun Jun 19 23:03:15 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Sun, 19 Jun 2022 23:03:15 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
Message-ID: <046401d88452$49aa1e60$dcfe5b20$@gmail.com>

With all due respect, Manprit, your text file may not end in .csv or .dat
but is just as structured as files that are. Many such files now have no
funny headers or anything and can be edited using a flat text editor of any
kind and you can enter data one at a time on each line with your designated
delimiter (space, tab, comma, colon, whatever) between items and not at the
end of a line. If all lines have the same number 
Of items and you save the file, you can often read it in no matter what the
extension is. 

Nobody is arguing with you about this, but trying to educate you. Just about
any programming language, not just python, has available facilities to read
in such files for you or let you read it a line at a time and use some
built-in method to split the line by those separators. 

If it turn out that your problem in reading it in using SQL or python is
that the name of the file ends in .txt, then do something to tell it what it
is in a way that over-rides the guess it makes OR rename or copy the file
(you can even do that in python) from xyy.txt to xyz.tsv or xyz.tab or
whatever makes the software happy!

I repeat. Most such files are indistinguishable from text files albeit there
may be some that add some kind of meta-data as something like comments that
software can use or ignore.

Most of the functionality people use to read in a file either lets you
specify that you expect the first line to be a header or that you do not
want it to be read. Some make a guess if the first line is not numeric and
the rest are. 

In your case, I believe you have an answer already and maybe have not
figured it out.

If you want us to write the code for you, hire someone. We are here more for
hints and education.

But in English, you seem to want a multistep process and to do it YOURSELF.
Fine. Write your function and call it ONCE like this:

Col1Name, col2Name <- func("file.xt")

You now swallowed one line of header and have the text,

THEN you can call your other function and let it call func() in iteration
till done. Every call will return a tuple of strings that contain a number
as text. 

You now have the names in one place and a growing list of "values" in
another. Presumably the way you call other code results in one place asking
to create or use a table with those column names, and the second keeps
inserting entries into that table by supplying data in the same order as the
columns.

I pointed out earlier that the code may not be right unless you made a
spelling mistake in showing it to us and you should read one line at a time
IN THE LOOP and not read one line once before. I am not convinced your
function is correct.

I am curious what you consider a big file. If your machine has enough
memory, you can process quite a large file and since python has garbage
collection, you can easily reclaim it by something as simple as putting
something new into the variable that held it, including deleting it
entirely. 

I find your suggestion about python confusing. Your application is
straightforward to do in just about ANY language I have used and that is
well over a hundred I have studied. Feel free to use another, but NOT
because you claim it is hard in python. Mind you, ideas like generators are
not everywhere you look. In some more limited languages, you might simply
read a line at a time and perhaps open a temporary file in which you place
SQL statements to start the process, then in a loop read your .txt and
produce a line or two vaguely like this:

INSERT INTO table_name (column1, column2, column3, ...)
VALUES (value1, value2, value3, ...);

Finally you could throw that file at your SQL provider and wait a week for
it to finish.

Many SQL databases allow you to put in many at once with some syntax like
this:

INSERT INTO MyTable
  ( Column1, Column2, Column3 )
VALUES
  ('John', 123, 'Lloyds Office'), 
  ('Jane', 124, 'Lloyds Office'), 
  ('Billy', 125, 'London Office'),
  ('Miranda', 126, 'Bristol Office');


You can imagine making those by yourself.

But you are using some modules that do this for you and that probably is a
great idea.  So work with it and see what the manual says is available. I
took a look as I have never had reason to use what you are talking about and
see your code:

cur.execute("create table workdata(Time INT, Pieces INT)")

The above is doing nothing but passing along what you created exactly. So
what you seem to want to do is take a string of text and interpolate the
words "Time" and "Pieces" into a template after reading them from the first
line of the text file, right? Python has at least 5 major ways to do that
and quite a few more. If you know how, fine. If not, ask a specific
question, please.

The next line of your code is:

cur.executemany("insert into workdata values (?, ?)",
data_generator("abcd.txt"))

The above is actually fairly simple shorthand for doing this by yourself in
pseudocode AFTER the table has been created using the first line of the
data-file:

Forever until out of data:
  Col1, Col2 = next_line_read_split
  MyCommand = interpolate Col1 and Col2 into "insert into workdata values
(%s, %s)" using one of many methods such as replacing the two instances of
%s
Run this: cur.execute(MyCommand)

The more complex loop you feed a generator to can be easily avoided if that
seems more complex. Heck, you do not need to have a generator at all, just
keep reading a line till done in the loop.

If the above makes no sense to you, fine. Ask specifically what step you do
not know how to do or give an example of your code and ask why what you did
is not working. Note the problem in cases like this often is not in the code
where you think but maybe in the code it calls.

If you want to solve a problem, this one is not hard in any number of ways.
If you want to solve it using cute shortcuts like the above that ask for a
generator, it may be a tad harder.


-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Manprit Singh
Sent: Sunday, June 19, 2022 9:00 PM
To: tutor at python.org
Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file

Dear all,
Learning is a lifelong process . At this point i still do not feel i am
absolutely ok with python or programming. In the past i was very active in
this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber,
dn and few others, they always answered my queries .
As i wrote in the starting , i have a simple text file named abcd.txt. (I am
not using a csv file or any other structured file) . The contents are given
below :

Time  Pieces
1       10
2       15
3       25
4       31
5       40
6       45
7       53
8       54
9       65
10      75

We can clearly see the first line of the file is

Time  Pieces

Now I have to read this text  into an sqlite3 database table , So it is
clear that the database table will have 2 columns, Column names of the
sqlite3 table are to be picked from the  first line of the text file
abcd.txt. (First column name will be Time, second column name will be Pieces
).
See i know this can be done very easily using numpy and Pandas. But I am
trying to do it with Core Python only.

cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))

I just want to insert column names (where these column names are  read from
text file abcd.txt 1st line)  using a question mark style placeholder.  Pls
let me know if anything is possible .

Time  Pieces

On Mon, Jun 20, 2022 at 5:34 AM <avi.e.gross at gmail.com> wrote:

> Manprit,
>
> You may be approaching this topic differently than you could.
>
> What you keep calling a TEXT file is what others might call structured 
> files like .CSV or .DAT types that contain values separated by some 
> delimiter like a comma or a tab or arbitrary whitespace or other 
> choices. Lots of python functionalities are available to read  these 
> in to structures used in modules like numpy and pandas that many 
> people use on top of or alongside regular built-in python commands. In 
> these, there can be (and usually are) HEADER names for columns and 
> sometimes also rows. The functionality that reads in such tables, 
> Dataframes, or whatever you want to call it, will often optionally 
> take the first line of the file to have headers specifying names. And, 
> you can always add or change the names as well.
>
> So if passing the entire structure to another function, it can choose 
> to access the name of any column (or often row) when it wants. Or, if 
> you need to, you can ask for it and pass it your way as in 
> interpolating the name into an SQL command.
>
> You may be asking a slightly different question here about just your data.
> Does your "text" file contain a first line saying ""Time        Pieces" or
> not?
>
> If it DOES, you may need to read it ONCE at the start before giving 
> your iterator for the numbers to be read and store the results as your 
> names to use as you wish. If the file has no header, then your code is 
> the one attaching a name.
>
> To me the real issue is your design seems to be of a limited nature. 
> You want to read in your data in a small batch (single line) and 
> presumably are opening a connection to the database and doing login 
> and so on, then sending a command to create a table, unless it already 
> exists, then sending umpteen commands to insert yet another row into 
> the table and then who knows? This seems like such a common operation 
> that I have to guess there is a way to do it fairly trivially and my 
> guess is way more efficiently.
>
>
>
> -----Original Message-----
> From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf 
> Of Manprit Singh
> Sent: Sunday, June 19, 2022 2:38 PM
> Cc: tutor at python.org
> Subject: Re: [Tutor] make a sqlite3 database from an ordinary text 
> file
>
> Dear sir ,
>
> One more query ....
> How can i use column name in text file as colum names in sqlite3 database
.
> I tried using python but failed.
>
> Can you put some light over it ?
> Regards
> Manprit Singh
>
> On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, <tutor at python.org>
> wrote:
>
> > On 19/06/2022 18:11, Manprit Singh wrote:
> >
> > > cur.execute("create table workdata(Time INT, Pieces INT)")
> >
> > This will fail the second time it runs. You should check if the 
> > table already exists and either drop it or use the existing table.
> >
> > However, I rarely create tables from Python, it's generally easier 
> > to create a sql file and run that directly using the sqlite interpreter.
> > You only need Python if the format of the table has to be deduced 
> > from the data. and that then leads to all sorts of follow-on issues 
> > over accessing names of columns etc.
> >
> > Once you have the table you can then use Python to load the data, 
> > although in your case you could do that directly from SQL too since 
> > it looks like a tab separated file.
> >
> > As always use the most appropriate tool. If you can do it directly 
> > in SQL there is no point in wrapping it in Python. That just adds 
> > complexity and slows things down.
> >
> > --
> > Alan G
> > Author of the Learn to Program web site http://www.alan-g.me.uk/ 
> > http://www.amazon.com/author/alan_gauld
> > Follow my photo-blog on Flickr at:
> > http://www.flickr.com/photos/alangauldphotos
> >
> >
> > _______________________________________________
> > Tutor maillist  -  Tutor at python.org
> > To unsubscribe or change subscription options:
> > https://mail.python.org/mailman/listinfo/tutor
> >
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avi.e.gross at gmail.com  Mon Jun 20 12:46:55 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Mon, 20 Jun 2022 12:46:55 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <t8pke5$jp3$1@ciao.gmane.io>
 <CAO1OCwYOK7Q4u+vNYat1cJaY-kF5t_H-8+SC5au8Nbkb1A4AAQ@mail.gmail.com>
 <CAO1OCwZtbDgiOij8__sThdvgTMAito_Q7fvozDC3kwjK4j0g1w@mail.gmail.com>
Message-ID: <012b01d884c5$5a1c0880$0e541980$@gmail.com>

Manprit,

When something is not working, think about it before asking others to.

The variable called "heads" is what? Try printing it. 

It came from " yield line.split()" and it probably is a list containing two
items of text as in ["title1", "title2"]

So verify what is in it using print() or other tools before continuing.

Then look to see what your cur.execute() wants there. Does it want one
argument that is a list or does it want two individual arguments?

OR, does it need any arguments there if you DO IT YOURSELF!

What if you made a line for yourself I which you interpolated heads[1] and
heads[2] into a string and executed that?

A little search shows examples like:

cur.execute("select md5(?)", (b"foo",))

cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)


Forget the Select part and focus on how it puts something dynamically into
the string.

See if you can make your code in a way that passes the right thing. The
second example above is one of many ways of interpolating by replacing the
%s within quotes v=by the value of "symbol" and you have two things to
replace.

Time to go live my real life while you do what seems like homework.




-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Manprit Singh
Sent: Monday, June 20, 2022 10:35 AM
To: tutor at python.org
Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file

 Dear Sir ,



 cur.execute("create table DATA(? INT, ? INT)",heads)

As in earlier mail I pointed out that the above command  will not work
because column names are not written inside the  single quotes in the
command create table.  Hence we can not use (?} place holders  .

Regards
Manprit Singh

On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh <manpritsinghece at gmail.com>
wrote:

> Dear Sir,
>
> Many thanks for this . This mail has a lot of things for me . I Will 
> do all the exercises on my own and will revert you soon.
>
> Regards
> Manprit Singh
>
> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor 
> <tutor at python.org>
> wrote:
>
>> On 20/06/2022 01:59, Manprit Singh wrote:
>>
>> > clear that the database table will have 2 columns, Column names of 
>> > the
>> > sqlite3 table are to be picked from the  first line of the text 
>> > file abcd.txt. (First column name will be Time, second column name 
>> > will be Pieces ).
>> > See i know this can be done very easily using numpy and Pandas. But 
>> > I am trying to do it with Core Python only.
>> >
>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", 
>> > "Pieces"))
>> >
>> > I just want to insert column names (where these column names are  
>> > read from text file abcd.txt 1st line)  using a question mark style 
>> > placeholder.
>>
>> OK, Having shown how I would really do it, I'll now try to answer the 
>> question for pure Python:
>>
>> def get_data(open_file):
>>     for line in open_file:
>>         yield line.split()
>>
>>
>> # open the database/cursor here....
>>
>> with open('data.txt') as inf:
>>     heads = inf.readline().split()
>>     cur.execute("create table DATA(? INT, ? INT)",heads)
>>     cur.executemany("insert into workdata values (?, 
>> ?)",get_data(inf))
>>
>> cur.execute("select * from DATA")
>> for row in cur.fetchall():
>>     print(row)
>>
>>
>> Note, I didn't actually run this but I think it should be close.
>>
>> --
>> Alan G
>> Author of the Learn to Program web site http://www.alan-g.me.uk/ 
>> http://www.amazon.com/author/alan_gauld
>> Follow my photo-blog on Flickr at:
>> http://www.flickr.com/photos/alangauldphotos
>>
>>
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From roel at roelschroeven.net  Mon Jun 20 04:37:45 2022
From: roel at roelschroeven.net (Roel Schroeven)
Date: Mon, 20 Jun 2022 10:37:45 +0200
Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file
In-Reply-To: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>
 <t8nonu$103i$3@ciao.gmane.io> <007a01d88436$58d53da0$0a7fb8e0$@gmail.com>
Message-ID: <e2bea4f4-263d-284f-3e7d-628cce1a8615@roelschroeven.net>

Op 20/06/2022 om 1:43 schreef avi.e.gross at gmail.com:
> I may not be looking at this right since much happens in functions not seen
> but does cur.execute(many) expect two items of text or two items of type
> integer or will it handle a list or tuple with arbitrary numbers of contents
> and so on, is what makes the design decision.
cur.executemany() takes an iterable which on each iteration yields a 
number of items corresponding to the number of placeholders ("?") in the 
query. You can use any datatype that sqlite3 understands, but care 
should be taken to use the type you need. Other than other databases, 
sqlite3 stores data as the type you give it, rather than the type 
specified in the database definition (in a sense sqlite3 is dynamically 
typed like Python, instead of statically typed like other SQL 
implementations).

So AFAICS the code is wrong on that point: it should convert the data to 
integers before handing them over to the database.

> Deeper within that function, it presumable maps your passed function to code
> that may do,
>
> First, Second = f()
>
> And so on. Or it may just take a single result or other choices. It is a bit
> confusing to pass a filename as someone pointed out.
This cur.executemany() call is the equivalent of:

 ??? for time, nr_pieces in data_generator('abcd.txt'):
 ??????? cur.execute("insert into workdata values (?, ?)", (time, 
nr_pieces))

But a bit shorter and with presumably less overhead.
> In what follows, either I am confused or the one asking the question is.
>
> The goal of the iterator seems to be to avoid reading all the data in at
> once, right. Otherwise, why bother when you can do something like reading in
> the entire file and splitting and putting the results in something like a
> list structure, or perhaps in something from numpy.
>
> So why does the code use readline() (no 's') to read in a single line and
> only once?
>
> As I read the somewhat confusing code, the goal is to open the file the
> first time and read a line and throw the results away as it is not clear
> where it is saved. Then the goal is to loop on nothing (or maybe one line)
> and yield the results of splitting it and pause till called again. But since
> only no or maybe one line have been received, the call to iterate should
> just exit as the loop is done.
>
> So I think reading a single line should be moved down within the loop!
That readline() call is to skip the first line, because it contains 
column names instead of actual data. After that the code iterates over 
the file object which yields each remaining line on each iteration, 
which is then split and yielded for use by cur.executemany().
> And you need some kind of test to know when there are no remaining lines in
> the file or the current line is empty and instead of yielding a non-existent
> result, you should return what the iteration protocol needs and close the
> file, albeit the with does that for you silently as part of that protocol.
No need, that's all taken care of automatically. "for line in obj" stops 
if there are no more lines, and as you say the with statement closes the 
file.
> But as others have noted, it is not clear other than as an exercise, if this
> iterator is needed. Your other modules may supply something for you.
Why not use an iterator? Once you get the hang of it, using iterators is 
often just as easy as using lists; with the advantage that a program 
that uses iterators where it matters works for small and large datasets 
without any problem because it doesn't need large amounts of memory to 
store its data.

-- 
"Peace cannot be kept by force. It can only be achieved through understanding."
         -- Albert Einstein


From roel at roelschroeven.net  Mon Jun 20 04:45:06 2022
From: roel at roelschroeven.net (Roel Schroeven)
Date: Mon, 20 Jun 2022 10:45:06 +0200
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
Message-ID: <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>

Op 20/06/2022 om 4:39 schreef dn:
> I'm +1 after @Alan. Although we often see it in programming texts, I've
> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS
> and SQL very early in my career, and therefore know to use the tools
> from that eco-system. (the rationale in books is the exact opposite -
> presuming the reader has never used DB-tools). Use the best tool for the
> job!
I don't really agree with this. I quite often use Python to read data 
from some file, perhaps process it a bit, then store it in SQL. If any 
kind of processing on the data is needed, Python is much easier and 
flexible to do it than SQL (at least to me). And if no processing is 
needed, well, I often use a Python script anyway because I know how to 
do that; doing it using the RDBMS tools would involve me starting to 
look up things left and right, and would be different between different 
RDBMS systems.

-- 
"Peace cannot be kept by force. It can only be achieved through understanding."
         -- Albert Einstein


From alan.gauld at yahoo.co.uk  Mon Jun 20 14:01:15 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 20 Jun 2022 19:01:15 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
 <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
Message-ID: <t8qclb$9ci$1@ciao.gmane.io>

On 20/06/2022 09:45, Roel Schroeven wrote:

>> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS
>> and SQL very early in my career, and therefore know to use the tools
>> from that eco-system. (the rationale in books is the exact opposite -
>> presuming the reader has never used DB-tools). Use the best tool for the
>> job!
> I don't really agree with this. I quite often use Python to read data 
> from some file, perhaps process it a bit, then store it in SQL. 

Roel,

We weren't saying not to use Python for processing SQL data.
It was specifically about creating the schema in Python
(and to a lesser degree loading it with initial data). You
can do it but it's usually much easier to do it directly
in SQL (which you have to embed in the Python code anyway!)

Once you have some data in a database using Python to process
the data makes sense.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From wlfraed at ix.netcom.com  Mon Jun 20 15:55:23 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Mon, 20 Jun 2022 15:55:23 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com> <t8qav4$pa8$1@ciao.gmane.io>
Message-ID: <28j1bhhdrgrrc860lblqn974sarmrejd1l@4ax.com>

On Mon, 20 Jun 2022 18:32:20 +0100, Alan Gauld via Tutor <tutor at python.org>
declaimed the following:

>On 20/06/2022 17:18, Dennis Lee Bieber wrote:
>
>> 	The SQLite3 adapter appears to inherently block using parameterized
>> queries to modify the database schema. If you REALLY must generate the
>> schema using data from a file, you have to do all of the that in Python,
>
>Ah! I didn't know that. Precisely because I never use Python
>to create a database!
>

	I'm not ambitious enough to try testing with Firebird or SQL-Server to
see if they object.

	I suspect SQLite3 probably objects to any DDL statement that is
parameterized -- I think it goes through the "prepare" (compile the SQL
statement)/"execute" (run the compiled statement with parameters). Schema
entities (database, table, fields...) probably have to be KNOWN during the
prepare/compile phase.

	One of the older MySQL adapters did not use "prepared statements" --
and internally it used Python string interpolation (with %s placeholders)
to plug in data values (and .executemany() might have been a loop issuing
statements for each set of data values). On that adapter, the OP's attempt
to define the schema from data values probably worked. As I recall, MySQL
in the 3.x series did not support prepared statements even from C. The
latest versions do support prepared statements (and other complex features)
and newer adapters may take advantage of such.

>That's true, although I might not say NEVER, since often you write
>one-off scripts for database work that nobody else ever uses. But the
>official mechanism works well enough for most real cases that there's
>rarely any reason not to use it.
>
	My throw-away efforts still have me looking at the input data file to
determine field names and data types (though SQLite3 doesn't really care --
if a field is defined INT and the data is a string that can be interpreted
as INT it converts, otherwise it will happily store it as text).

>And creating tables and columns dynamically is frought with problems
>too. It's like trying to create variables in code from user input - how
>does the rest of the code refer to those values? For variables we can
>use a dictionary as a partial solution but for a database it gets very
>messy with the need to look at the meta data before performing any action.

	And that leads back to my prior post about "menus". If the application
needs the user to specify fields for matching in a WHERE clause, say, I'd
present a menu of field names obtained from the schema, and the user would
select by menu position, not by entering the name. That way I know the
field name(s) can not contain garbage, and it is safe to use Python string
interpolation to build the SQL statement, while using parameterization for
the user provided values to be matched.



-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From roel at roelschroeven.net  Mon Jun 20 15:18:25 2022
From: roel at roelschroeven.net (Roel Schroeven)
Date: Mon, 20 Jun 2022 21:18:25 +0200
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <t8qclb$9ci$1@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
 <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
 <t8qclb$9ci$1@ciao.gmane.io>
Message-ID: <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net>



Alan Gauld via Tutor schreef op 20/06/2022 om 20:01:
> On 20/06/2022 09:45, Roel Schroeven wrote:
>
> >> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS
> >> and SQL very early in my career, and therefore know to use the tools
> >> from that eco-system. (the rationale in books is the exact opposite -
> >> presuming the reader has never used DB-tools). Use the best tool for the
> >> job!
> > I don't really agree with this. I quite often use Python to read data 
> > from some file, perhaps process it a bit, then store it in SQL. 
>
> Roel,
>
> We weren't saying not to use Python for processing SQL data.
> It was specifically about creating the schema in Python
> (and to a lesser degree loading it with initial data). You
> can do it but it's usually much easier to do it directly
> in SQL (which you have to embed in the Python code anyway!)

Ah, I misunderstood. My apologies.

-- 
"Experience is that marvelous thing that enables you to recognize a
mistake when you make it again."
         -- Franklin P. Jones


From mats at wichmann.us  Mon Jun 20 17:22:12 2022
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 20 Jun 2022 15:22:12 -0600
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <t8qclb$9ci$1@ciao.gmane.io>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
 <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
 <t8qclb$9ci$1@ciao.gmane.io>
Message-ID: <c78e0465-a749-a8d1-dc32-c7d71ec21c34@wichmann.us>

On 6/20/22 12:01, Alan Gauld via Tutor wrote:
> On 20/06/2022 09:45, Roel Schroeven wrote:
> 
>>> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS
>>> and SQL very early in my career, and therefore know to use the tools
>>> from that eco-system. (the rationale in books is the exact opposite -
>>> presuming the reader has never used DB-tools). Use the best tool for the
>>> job!
>> I don't really agree with this. I quite often use Python to read data 
>> from some file, perhaps process it a bit, then store it in SQL. 
> 
> Roel,
> 
> We weren't saying not to use Python for processing SQL data.
> It was specifically about creating the schema in Python
> (and to a lesser degree loading it with initial data). You
> can do it but it's usually much easier to do it directly
> in SQL (which you have to embed in the Python code anyway!)

This isn't as true (especially the "have to embed sql anyway") if you're
using an ORM.



From alan.gauld at yahoo.co.uk  Mon Jun 20 18:55:02 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 20 Jun 2022 23:55:02 +0100
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <c78e0465-a749-a8d1-dc32-c7d71ec21c34@wichmann.us>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
 <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
 <t8qclb$9ci$1@ciao.gmane.io>
 <c78e0465-a749-a8d1-dc32-c7d71ec21c34@wichmann.us>
Message-ID: <t8qts6$eni$1@ciao.gmane.io>

On 20/06/2022 22:22, Mats Wichmann wrote:

>> We weren't saying not to use Python for processing SQL data.
>> It was specifically about creating the schema in Python
> 
> This isn't as true (especially the "have to embed sql anyway") if you're
> using an ORM.

Thats true, but then an ORM is there to pretend that a SQL
RDBMS is actually an OODB. So using SQL for anything is
likely to break stuff because the ORM controls how the
relationships are set up and mapped.

But I confess I've rarely used ORMs, I'd rather just use
a real Object Database if I need to maage persistent objects.
But I' haven't done much of that either, in fact the only
OODB I'd say I'd recommend is Objectivity, although I'd
like to have a play with ZODB someday.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From phillor9 at gmail.com  Tue Jun 21 00:17:06 2022
From: phillor9 at gmail.com (Phil)
Date: Tue, 21 Jun 2022 14:17:06 +1000
Subject: [Tutor] Event loop logic question
In-Reply-To: <ls7sahpm4ro412kbap6b7ee6euamur51o7@4ax.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <ls7sahpm4ro412kbap6b7ee6euamur51o7@4ax.com>
Message-ID: <f30ca141-cd74-02ab-2f00-925bff28e488@gmail.com>


On 19/6/22 05:19, Dennis Lee Bieber wrote:
> On Sat, 18 Jun 2022 15:04:50 +1000, Phil <phillor9 at gmail.com> declaimed the
> following:
>
>
>> Any hints or alternative methods will be, as always, greatly appreciated.
>>
>> By the way, this project sends commands to a controller which is easily
>> overwhelmed if too many commands are received per second.
>>
Thank you avi, Alan and Dennis for your time.

I've been in an area without mobile phone reception for the past couple 
of days and it could be awhile before I'm in an area with reasonable 
reception again. In the meantime I'll give some thought to the 
suggestions offered and see what I can come up with. The controller 
firmware is known to contain bugs (crashing because of commands in quick 
succession is one of them).

The print statements at seemingly random places are just my feeble 
attempt at debugging the code.

-- 

Regards,
Phil


From avi.e.gross at gmail.com  Mon Jun 20 18:02:42 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Mon, 20 Jun 2022 18:02:42 -0400
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <uj51bhtesg2hm2kopsu15u2tvvvupp24gf@4ax.com>
Message-ID: <018201d884f1$778aa7b0$669ff710$@gmail.com>

Dennis,

While your comments are quite valid and reasonable and make me wonder about
all sorts of techniques hackers can use that often must be defended against,
wouldn't it be nice if people doing homework would actually show us the
problem rather than trying to give a few parts?

If this is meant to be serious code to be heavily used in real world
situations, such as for a job at Google, it does often require some
bulletproofing as you describe. But is it that or just solving a problem I a
sandbox to learn how to use a few features, or is it more akin to homework?

I am not saying the case in point is homework but it has some signs I
interpret that way. It may be the kind of homework where you do independent
study from a book and are trying the questions at the end of the chapter and
really want to learn, not where you want to get someone to just do it for
you in a class and not learn to do it yourself.  It seems to both be open to
some packages and advanced ideas while we get steered away from others. Some
of the replies probably are way off base if we only knew the context.

I had to go out for the afternoon at this point and note that with amusement
that a book I was reading about web development happened to cover various
ways to store and query data from a browser on the local machine and had a
section where a similar javascript interface tp something like SQL LITE was
used that clearly showed how the API expected multiple names to be given as
a single argument with something like a list. So I am not sure why our
persistent questioner said the code he tried did not work.

I look for patterns and anomalies that might help me GUESS things and I saw
some here that may, and may not, mean anything.

For example, why is it required to use a text file with what looks like a
space or tab or whitespace separator? Alan has suggested ways to 
Change the file to something like a CSV that can be done directly in the
non-python arena of the database. Well, yes, that is a perfectly reasonable
approach if you are not told the goal is to do it all in a circumscribed
subset of python!

In real life, other than perhaps efficiency considerations, many people
would start with existing tools such as numpy and pandas and read the darn
data in natively,  specifying a delimiter as in:

pd.read_csv('file.csv', delimiter=' ')

is a fairly easy way to read his "text file" unless a requirement is made to
read it in a line at a time.  Functions like the above typically read quite
a bit of the file to determine what types the columns of data should be, so
they probably are not designed to act much like generators. But I suspect
they could easily be if the goal was not to read in all of the file. You
could read in a thousand lines or so and serve those until you ran low and
got another batch and so on. 

So is it really a requirement to read a line at a time and not to make sure
all columns are consistent? Apparently not.

Nor were we told of any requirements that this be SQL safe. Yes, tools can
ensure that but if this was a beginners course, I might add requirements
like take the two text items you got from the split and checks them several
ways before using them. They should all be digits, and no longer than some
length of say 8 characters. If you allow commas or underscores, strip them.
If they can be decimals, or scientific notation, it gets more complex but
sometimes something as simple as calling a function that converts the
contents to a number and then converting them back, can reject or repair
many such strings and possibly make them safe to use by interpolating the
contents directly.

I can not be sure but I think the main problem in this example was poor
programming based on what may not have been a well-planned design. 

Loops need some things done before and some within and some after and you
need to segregate parts of your plan so each is done as often as needed
(such as once) and in the right place. 

In my view, this assignment of self-motivate project is not that complicated
if approached systematically with a knowledge of how things can be done and
some minor skills at debugging such as strategically placed print statement.
The fact that several of us are going back and forth may partially be due to
some English language issues and might work better if we communicated in
their native language. I suspect none of us participating does! LOL! I do
not speak Punjabi or even Hindi. So patience and some awareness the other is
working hard to try to communicate is necessary. 

Where are we? Several possible methods have been offered using
advice/pseudocode or actual python code and if none of them works, we need
more than just being told the user could not get it to work. WHERE did it
fail? What were the context of variables? What error message? Did they
cut/paste and adjust indentation correctly, or perhaps type something
directly but wrong?

There is only so much time and effort being offered here and it is, in my
case, more about teaching someone so they can then do it again on their own,
than producing unpaid work or something that shows off what we can do but
not what they need.

-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of
Dennis Lee Bieber
Sent: Monday, June 20, 2022 12:18 PM
To: tutor at python.org
Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file

On Mon, 20 Jun 2022 06:29:58 +0530, Manprit Singh
<manpritsinghece at gmail.com> declaimed the following:

>See i know this can be done very easily using numpy and Pandas. But I 
>am trying to do it with Core Python only.
>
>cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces"))
>
>I just want to insert column names (where these column names are  read 
>from text file abcd.txt 1st line)  using a question mark style 
>placeholder.  Pls let me know if anything is possible .
>

	You DON'T!

	As stated in my previous post, that is manipulating the database
schema
-- which should never rely upon values found in data files due to the
possibility of an SQL injection attack.

https://www.explainxkcd.com/wiki/index.php/327:_Exploits_of_a_Mom

	The purpose of using the parameterized placeholders is that the
interface can "sanitize" (escape/quote) characters in the DATA that have
meaning in SQL.

	The SQLite3 adapter appears to inherently block using parameterized
queries to modify the database schema. If you REALLY must generate the
schema using data from a file, you have to do all of the that in Python,
BEFORE passing it to .execute().

>>> import sqlite3 as db
>>> con = db.connect("junk.db")
>>> cur = con.cursor()
>>> cur.execute("create table workdata (? INT, ? INT)", ("Time", 
>>> "Pieces"))
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
sqlite3.OperationalError: near "?": syntax error
>>> cur.execute("create table workdata (%s INT, %s INT)" % ("Time", 
>>> "Pieces"))
<sqlite3.Cursor object at 0x000002B642A8D030>
>>> cur.execute("insert into workdata (Time, Pieces) values (?, ?)", 
>>> (123, "who"))
<sqlite3.Cursor object at 0x000002B642A8D030>
>>>

	Note that the second .execute() is using Python string interpolation
to create the schema definition SQL, not parameterized SQLite3 operations.
The third .execute() shows that parameterized statements work with DATA.
And, per the warning about SQL injection attacks, you should NEVER use
string interpolation (or other formatting operations) in Python to work with
data for the database.


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avi.e.gross at gmail.com  Mon Jun 20 18:18:15 2022
From: avi.e.gross at gmail.com (avi.e.gross at gmail.com)
Date: Mon, 20 Jun 2022 18:18:15 -0400
Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file
In-Reply-To: <e2bea4f4-263d-284f-3e7d-628cce1a8615@roelschroeven.net>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <CAO1OCwaJAQ_d2ziL27yY6b6W5uSt+gdtPwVuy+EJ0yH5t+OH8w@mail.gmail.com>
 <t8nonu$103i$3@ciao.gmane.io> <007a01d88436$58d53da0$0a7fb8e0$@gmail.com>
 <e2bea4f4-263d-284f-3e7d-628cce1a8615@roelschroeven.net>
Message-ID: <01a801d884f3$a3874420$ea95cc60$@gmail.com>

Roel,

I appreciate your explanations.

I wonder about one thing here. From your description, it sounds like the fully-text file being read needs to have the first line read in as text and remain as text. But in this case, all subsequent files should be sent to the database as integers or perhaps another numeric format. 

If that is correct, the generator used normally cannot return both of those easily. For this one case, you could open the darn file once to get the column names without using the generator, to get the column names as text, then reopen the file in the generator and throw away the first result as the generator will now return an integer form of the data  and the first line will be nonsense. Several variations are possible, and since a generator can keep state, it can be set to effectively skip the first line. 

The designs so far have not shown any idea of making integers, albeit a wrapper around the generator could perhaps do the conversion.

Some of this makes me appreciate all the hidden level of detail that read_csv() and similar functions deal with that may not be needed every time or may not be perfect, but can be relied on to deal with lots of detail you might not even think about.


-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of Roel Schroeven
Sent: Monday, June 20, 2022 4:38 AM
To: tutor at python.org
Subject: Re: [Tutor] Fwd: make a sqlite3 database from an ordinary text file

Op 20/06/2022 om 1:43 schreef avi.e.gross at gmail.com:
> I may not be looking at this right since much happens in functions not 
> seen but does cur.execute(many) expect two items of text or two items 
> of type integer or will it handle a list or tuple with arbitrary 
> numbers of contents and so on, is what makes the design decision.
cur.executemany() takes an iterable which on each iteration yields a number of items corresponding to the number of placeholders ("?") in the query. You can use any datatype that sqlite3 understands, but care should be taken to use the type you need. Other than other databases,
sqlite3 stores data as the type you give it, rather than the type specified in the database definition (in a sense sqlite3 is dynamically typed like Python, instead of statically typed like other SQL implementations).

So AFAICS the code is wrong on that point: it should convert the data to integers before handing them over to the database.

> Deeper within that function, it presumable maps your passed function 
> to code that may do,
>
> First, Second = f()
>
> And so on. Or it may just take a single result or other choices. It is 
> a bit confusing to pass a filename as someone pointed out.
This cur.executemany() call is the equivalent of:

     for time, nr_pieces in data_generator('abcd.txt'):
         cur.execute("insert into workdata values (?, ?)", (time,
nr_pieces))

But a bit shorter and with presumably less overhead.
> In what follows, either I am confused or the one asking the question is.
>
> The goal of the iterator seems to be to avoid reading all the data in 
> at once, right. Otherwise, why bother when you can do something like 
> reading in the entire file and splitting and putting the results in 
> something like a list structure, or perhaps in something from numpy.
>
> So why does the code use readline() (no 's') to read in a single line 
> and only once?
>
> As I read the somewhat confusing code, the goal is to open the file 
> the first time and read a line and throw the results away as it is not 
> clear where it is saved. Then the goal is to loop on nothing (or maybe 
> one line) and yield the results of splitting it and pause till called 
> again. But since only no or maybe one line have been received, the 
> call to iterate should just exit as the loop is done.
>
> So I think reading a single line should be moved down within the loop!
That readline() call is to skip the first line, because it contains column names instead of actual data. After that the code iterates over the file object which yields each remaining line on each iteration, which is then split and yielded for use by cur.executemany().
> And you need some kind of test to know when there are no remaining 
> lines in the file or the current line is empty and instead of yielding 
> a non-existent result, you should return what the iteration protocol 
> needs and close the file, albeit the with does that for you silently as part of that protocol.
No need, that's all taken care of automatically. "for line in obj" stops if there are no more lines, and as you say the with statement closes the file.
> But as others have noted, it is not clear other than as an exercise, 
> if this iterator is needed. Your other modules may supply something for you.
Why not use an iterator? Once you get the hang of it, using iterators is often just as easy as using lists; with the advantage that a program that uses iterators where it matters works for small and large datasets without any problem because it doesn't need large amounts of memory to store its data.

-- 
"Peace cannot be kept by force. It can only be achieved through understanding."
         -- Albert Einstein

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From PythonList at DancesWithMice.info  Tue Jun 21 05:07:37 2022
From: PythonList at DancesWithMice.info (dn)
Date: Tue, 21 Jun 2022 21:07:37 +1200
Subject: [Tutor] make a sqlite3 database from an ordinary text file
In-Reply-To: <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net>
References: <CAO1OCwYrjscsRB_1nmgzVSXnG6=thWDSdxN7RMe6GAJByMKJfA@mail.gmail.com>
 <t8noes$103i$1@ciao.gmane.io>
 <CAO1OCwZ13W0bnmGquskW+WG9GHKYkydu7MPT=R7sD-aU_dyvRA@mail.gmail.com>
 <007e01d88438$63156090$294021b0$@gmail.com>
 <CAO1OCwaf6m_V16mf9vYqu5to-4ZyEteBRQyNNFUGha5DYF_N1Q@mail.gmail.com>
 <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info>
 <c9c48391-6ec3-9cdd-5178-a211ffe3b382@roelschroeven.net>
 <t8qclb$9ci$1@ciao.gmane.io>
 <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net>
Message-ID: <f2fc6c8c-0392-41df-2ab6-7f9b6cfdca23@DancesWithMice.info>

On 21/06/2022 07.18, Roel Schroeven wrote:
> 
> 
> Alan Gauld via Tutor schreef op 20/06/2022 om 20:01:
>> On 20/06/2022 09:45, Roel Schroeven wrote:
>>
>> >> NEVER initialised a DB/TBL this way. Easy for me to say: I learned
>> RDBMS
>> >> and SQL very early in my career, and therefore know to use the tools
>> >> from that eco-system. (the rationale in books is the exact opposite -
>> >> presuming the reader has never used DB-tools). Use the best tool
>> for the
>> >> job!
>> > I don't really agree with this. I quite often use Python to read
>> data > from some file, perhaps process it a bit, then store it in SQL.
>> Roel,
>>
>> We weren't saying not to use Python for processing SQL data.
>> It was specifically about creating the schema in Python
>> (and to a lesser degree loading it with initial data). You
>> can do it but it's usually much easier to do it directly
>> in SQL (which you have to embed in the Python code anyway!)
> 
> Ah, I misunderstood. My apologies.

There's plenty of room for that, just as there is plenty of room for
using Python, SQL-tools, or something else. (I wasn't offended)

The different SQL eco-systems, eg DB2, Oracle, MySQL, SQLite; offer very
different ranges of tools beyond the RDBMS 'engine' and command-line
client. Thus, it is all-very-well for me/us to say 'use the SQL-tools'
if we are talking about one system and for you/A.N.Other to question
such if using something less feature-full.

Then there are the points you have made. If there is quite a bit of
'data-cleaning' going-on, then such will start to exceed the capability
of the tools I/we've espoused. In which case, Python is the 'Swiss Army
Knife' to use - no debate.

There are plenty of rules and warnings about sanitising data before
INSERT-ing or UPDATE-ing. Anyone unaware of that should stop coding and
do some reading *immediately*! That someone would dare to use user-data
to formulate a schema is far to rich (risky) for my blood! I'm trying to
imagine a reasonable use-case, but...

Forced into some sort of situation like this, I'd split the task into
parts, with a manual inspection of the generated schema before I'd let
it run. (and once again, I'd input that schema through an SQL-tool - but
allow for the fact that my background enables me to achieve such
quickly. Whereas YMMV and thus you/A.N.Other might prefer to use the
Python interface)

Another consideration worth some thought is that key-value databases are
'schema-less'. The fact that in this scenario we don't know what the
schema will be until the first phase of the system 'designs it', becomes
irrelevant! The other nifty advantage to using something like MongoDB is
that its mechanisms are virtually indistinguishable from Python dicts
(etc) - so I'll often prototype with Python built-ins and only later
worry about persistence.

-- 
Regards,
=dn

From tempest.amogh at gmail.com  Tue Jun 21 01:47:26 2022
From: tempest.amogh at gmail.com (tempest.amogh )
Date: Tue, 21 Jun 2022 11:17:26 +0530
Subject: [Tutor] Help with brute force approach to solving grid combinations
Message-ID: <a5dbe839-c42d-407b-847a-a0da7c25d21a@Canary>

As a personal project, I was inspired by Erik Demaine. He has created a font based on tetris shapes [http://erikdemaine.org/fonts/], however I would like to emulate this but based on another reference. I want to write an algorithm that brute force solves all the positions for these pieces in the english alphabet. I am having trouble making a grid that can be solved with these pieces, despite a book saying so.

I have attached the image that shows the shapes [http://erikdemaine.org/fonts/tetris/kadon_fonts.jpg].

My code:

---

import numpy as np

default_grid = np.array([[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]
])

a_grid = np.array([[0, 0, 0, 0, 0, 0],
[0, 0, -1, -1, 0, 0],
[0, 0, -1, -1, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, -1, -1, 0, 0],
[0, 0, -1, -1, 0, 0]
])

# 1. SHAPES

# 1.1 DOMINOES

shape_straight2 = np.array([[7, 0, 0, 0, 0, 0],
[7, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]
])

shape_flat = np.array([[0, 0, 0, 0, 8, 8],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]
])

# 1.2 TRIMINOES

shape_corner = np.array([[6, 0],
[6, 6]
])

shape_straight3 = np.array([[7],
[7],
[7]
])

# 1.3 TETRIMINOES

shape_straight = np.array([[1, 1, 1, 1]
])

shape_straight1 = np.array([[1],
[1],
[1],
[1]
])

shape_square = np.array([[2, 2],
[2, 2]
])

shape_l = np.array([[3, 0],
[3, 0],
[3, 3]
])

shape_l2 = np.array([[0, 3],
[0, 3],
[3, 3]
])

shape_l3 = np.array([[3, 3, 3],
[0, 0, 3]
])

shape_l4 = np.array([[3, 3, 3],
[3, 0, 0]
])

shape_skew = np.array([[4, 4, 0],
[0, 4, 4]
])

shape_skew2 = np.array([[4, 0],
[4, 4],
[0, 4]
])

shape_skew3 = np.array([[0, 4, 4],
[4, 4, 0]
])

shape_skew4 = np.array([[0, 4],
[4, 4],
[4, 0]
])

shape_t = np.array([[5, 0],
[5, 5],
[5, 0]
])

shape_t2 = np.array([[5, 5, 5],
[0, 5, 0]
])

shape_t3 = np.array([[0, 5],
[5, 5],
[0, 5]
])

shape_t4 = np.array([[0, 5, 0],
[5, 5, 5]
])

new_shape = np.array(default_grid + shape_t4)

---

I ran a test with just inserting one shape into a grid. But the following error messages comes up.

In the final code, I want there to be a way to check the coordinates and insert the shapes there will many for loops and if elif statements.

Error Message:
120, in <module>
new_shape = np.array(default_grid + shape_t4)
ValueError: operands could not be broadcast together with shapes (6,6) (2,3)

?
Regards,

Amogh Atwe


From phillor9 at gmail.com  Thu Jun 23 20:57:15 2022
From: phillor9 at gmail.com (Phil)
Date: Fri, 24 Jun 2022 10:57:15 +1000
Subject: [Tutor] Event loop logic question
In-Reply-To: <t8k80n$dqc$1@ciao.gmane.io>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <t8k80n$dqc$1@ciao.gmane.io>
Message-ID: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>

Problem solved.

I suddenly realised, while watching the output of the print statements, 
that I don't need the update function. Instead the control statements 
should be in the keyboard reading functions.

Advice for self; don't bother the mailing list but instead give the 
problem a rest for a day or two.

-- 
Regards,
Phil


From marcus.luetolf at bluewin.ch  Sat Jun 25 14:45:37 2022
From: marcus.luetolf at bluewin.ch (marcus.luetolf at bluewin.ch)
Date: Sat, 25 Jun 2022 20:45:37 +0200
Subject: [Tutor] problem solving with lists: final (amateur) solution
Message-ID: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>

Hello Experts, hello dn,
it's a while since I - in terms of Mark Lawrence - bothered you with my
problem.
Thanks to your comments, especially to dn's structured guidance I've come up
with the code below, based on repeatability.
I am shure there is room for improvement concerning pythonish style but for
the moment the code serves my purposes.
A commented version can be found on
https://github.com/luemar/player_startlist.

def startlist(all_players, num_in_flight):
    c_all_players = all_players[:]
    history = {'a':[], 'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],\
               'i':[],'j':[],'k':[],'l':[],'m':[],'n':[],'o':[],'p':[]}
    print('...............day_1................')
    def day_1_flights():
        key_hist = list(history.keys())
        c_key_hist = key_hist[:]
        for dummy_i in c_key_hist:
            print('flights_day_1: ', c_key_hist[:num_in_flight])
            for key in c_key_hist[:num_in_flight]:
                [history[key].append(player)for player in
c_all_players[0:num_in_flight]]
            del c_key_hist[:num_in_flight]
            del c_all_players[0:num_in_flight]
    day_1_flights()
    
    def day_2_to_5_flights():
        flights = {}
        for i in range(2,6):
            print('...............day_' + str(i)+'................')
            flights['a_flight_day_'+str(i)]= []
            flights['b_flight_day_'+str(i)]= []
            flights['c_flight_day_'+str(i)]= []
            flights['d_flight_day_'+str(i)]= []            
            lead = list('abcd')                   
            flight_list = [flights['a_flight_day_'+str(i)],
flights['b_flight_day_'+str(i)],\
                           flights['c_flight_day_'+str(i)],
flights['d_flight_day_'+str(i)]]

            for j in range(len(flight_list)):            
                def flight(cond, day):
                    for player in all_players:
                        if player not in cond:
                            day.extend(player)
                            cond.extend(history[player])
                            history[lead[j]].extend(player)
                    day.extend(lead[j])
                    day.sort()
                    [history[pl].extend(day) for pl in day[1:]]
                    return lead[j]+'_flight_day_'+str(i)+ ': ' +
str(flight_list[j])
                   
                conditions = [history[lead[j]], history[lead[j]] +
flights['a_flight_day_'+str(i)],\
                              history[lead[j]] +
flights['a_flight_day_'+str(i)] + \
                              flights['b_flight_day_'+str(i)], \
                              history[lead[j]] +
flights['a_flight_day_'+str(i)] + \
                              flights['b_flight_day_'+str(i)]+
flights['c_flight_day_'+str(i)]] 
                print(flight(list(set(conditions[j])), flight_list[j]))
    day_2_to_5_flights()
startlist(list('abcdefghijklmnop'), 4)

 Many thanks, Marcus.

-----Urspr?ngliche Nachricht-----
Von: Tutor <tutor-bounces+marcus.luetolf=bluewin.ch at python.org> Im Auftrag
von dn
Gesendet: Mittwoch, 25. Mai 2022 22:02
An: tutor at python.org
Betreff: Re: [Tutor] problem solving with lists: preliminary solution

Hi Marcus,

This is a bit of 'blast from the past' - and I haven't gone 'back' to look
at our previous correspondence/this message hasn't "threaded".

May I offer a few criticisms and suggestions:-


On 26/05/2022 05.19, marcus.luetolf at bluewin.ch wrote:
> ....sorry, forgott correct e_mail address.....
> 
> Hello Experts, hello dn,

and not forgetting @wlfraed who mentioned university research papers dealing
with the intricacies (and insolubles) of the "SGP". Did you try following-up
on any of those?


> In resuming my task to write code for a special constellation of the 
> "SocialGolferProblem" (SGP) :
> number_of_days  (number_of_weeks) = 5
> number_of_flights (number_of_groupings)  = 4 seize of flight 
> (players_per_grouping) = 4

Criticism: although each of these terms is perfectly understandable, and
easily converted into a 'readable' pythonic identifier, later the code uses
"n" (for example) - which of the two values commencing "*n*umber_"
is it?


> and given the solution below (draws for week 1 through 5) I've come up 
> with a code of which I only post part of it (for it gets to big):
> for day 1( hardcoded)  and for day 2 (as a function to be used through 
> day 5).

The word "hardcoded" immediately stopped me in my tracks!

The whole point of using the computer is to find 'repetition' and have the
machine/software save us from such boredom (or nit-picking detail in which
we might make an error/become bored).

That said, may I suggest that you grab a whiteboard/blackboard, or a
convenient wall and a bunch of Post-It notes, and try to solve the problem
manually - first for week-one (which is largely trivial), but then recording
the 'history' and moving-on to the extra decision-layer for week-two. If
your brain doesn't go 'click'*, go on to deal with week-three and see how
the algorithm develops...

* keeping my 'pocket handkerchief' lawn tidy doesn't require 'heavy
machinery', but after many years of service my little electric mower 'died'.
I bought a new Bosch model which needed to be partly assembled from the box.
The accompanying manual featured explanations in dozens of languages.
However, the (single/all-language) diagram showing where the back-wheels
were to be joined to the axles featured (only) the English
explanation: "Click". Perhaps other languages do use the word (?spelling),
but the term "going click in your mind" has long been used for the idea
explained in today's idiom as an "ahah! moment".


> The crucial part of my functions for day 2 to 5 are the conditional 
> statements. These conditional statements do not follow a consistent
pattern.
> This inconsistency casts some doubt on the effectiveness of my code 
> below and I'd appreciate critical comments.
> 
...
>     a_flight_day_1 = []
>     b_flight_day_1 = []
>     c_flight_day_1 = []
...
>     history = {'a':[],
>
'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],'i':[],'j':[],'k':[],'l':[]
,'m':[],'n':[],'o':[],'p':[]}
...

Following-on from talking about looping (repetition), yes we need to use
multiple conditional expressions to ensure that history is taken into
account (consistently).

The other 'side' of both of these code-constructs is the data-construct.
Code-loops require data-collections! The hard-coded "a" and "day_1" made me
shudder.
(not a pretty sight - the code, nor me shuddering!)

Would you benefit from spending a little time, putting aside the SGP for a
moment, and looking at a tutorial on "nesting" lists (and
dictionaries) in Python?


Again, the above-mentioned 'whiteboard' exercise may help 'reveal' the
patterns in the data, as well as identifying an algorithm.


Sadly, the 'hard-coded' parts may 'help' sort-out week-one, but (IMHO) have
made things impossibly-difficult to proceed into week-two (etc).

(and I'm back to mentioning that I haven't searched for our previous
discussions - specifically whether we talked-about a data-construct for
'history', and also to referring-back to the 'whiteboard' suggestion)


We were warned!

The SGP can be made to work for this particular combination of
weeks/flights/players. Which gives credence to the first-thought: "it looks
so easy". However, the wider problem is more complex than one at-first
imagines!

This is the reason why 'combinatorial problems' are so interesting. Did I
say "interesting"? Perhaps "frustrating" and/or "frustratingly difficult to
solve" - or even: 'cannot (always) be solved'!

This is no 'little kitten' or some old piece of rope, you have "a tiger by
the tail"...
--
Regards,
=dn
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From alan.gauld at yahoo.co.uk  Sat Jun 25 18:59:40 2022
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 25 Jun 2022 23:59:40 +0100
Subject: [Tutor] Event loop logic question
In-Reply-To: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <t8k80n$dqc$1@ciao.gmane.io> <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>
Message-ID: <t9840s$ab4$1@ciao.gmane.io>

On 24/06/2022 01:57, Phil wrote:
> Problem solved.
> 
> I suddenly realised, while watching the output of the print statements, 
> that I don't need the update function. Instead the control statements 
> should be in the keyboard reading functions.

I don't know your particular framework but as a general rule UI
changes should be in the paint/draw/update function and only
state data changes in the event handlers. So based purely on
general principles I'd say you are completely wrong and ALL
of the print tatements should be in the update and none of
them in the key hanling functions.

But that assumes that somewhere your framework calls update
regularly to redraw the UI. If that's not the case then all
bets are off. But if it is true you will save yourself an
awful lot of grief later by keeping the print statements
in one place.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From phillor9 at gmail.com  Sat Jun 25 20:06:25 2022
From: phillor9 at gmail.com (Phil)
Date: Sun, 26 Jun 2022 10:06:25 +1000
Subject: [Tutor] Event loop logic question
In-Reply-To: <t9840s$ab4$1@ciao.gmane.io>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <t8k80n$dqc$1@ciao.gmane.io> <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>
 <t9840s$ab4$1@ciao.gmane.io>
Message-ID: <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com>


On 26/6/22 08:59, Alan Gauld via Tutor wrote:
>
> I don't know your particular framework but as a general rule UI
> changes should be in the paint/draw/update function and only
> state data changes in the event handlers.

Thank you Alan once again for your time and your comments are always 
welcome.

I was looking for an easy way to repeatedly read the keyboard and 
pygamezero fills the bill. The problem that I initially had was that the 
update function is called many times per second and sending control 
commands from there caused the receiving hardware to be overwhelmed and 
then to crash. The UI is a 20 * 20 square and servers no use at all, 
it's just a requirement as far as I can tell, and so does not need 
updating. All I need is to read the arrow keys and to be able to quit 
the programme.

There is no need for any user output to be displayed on the transmitting 
laptop, the print statements were just a debugging aid. Briefly, I SSH 
from my laptop to a raspberry pi via my mobile phone created WiFi 
network. No doubt it's an amateurish solution, however, it's simple and 
does exactly what I had intended.

-- 

Regards,
Phil


From PythonList at DancesWithMice.info  Sat Jun 25 20:34:54 2022
From: PythonList at DancesWithMice.info (dn)
Date: Sun, 26 Jun 2022 12:34:54 +1200
Subject: [Tutor] problem solving with lists: final (amateur) solution
In-Reply-To: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
Message-ID: <c021a502-f4b8-232b-17b8-97abcc10eb86@DancesWithMice.info>

On 26/06/2022 06.45, marcus.luetolf at bluewin.ch wrote:
> Hello Experts, hello dn,
> it's a while since I - in terms of Mark Lawrence - bothered you with my
> problem.
> Thanks to your comments, especially to dn's structured guidance I've come up
> with the code below, based on repeatability.
> I am shure there is room for improvement concerning pythonish style but for
> the moment the code serves my purposes.
> A commented version can be found on
> https://github.com/luemar/player_startlist.
> 
> def startlist(all_players, num_in_flight):
>     c_all_players = all_players[:]
>     history = {'a':[], 'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],\
>                'i':[],'j':[],'k':[],'l':[],'m':[],'n':[],'o':[],'p':[]}
>     print('...............day_1................')
>     def day_1_flights():
>         key_hist = list(history.keys())
>         c_key_hist = key_hist[:]
>         for dummy_i in c_key_hist:
>             print('flights_day_1: ', c_key_hist[:num_in_flight])
>             for key in c_key_hist[:num_in_flight]:
>                 [history[key].append(player)for player in
> c_all_players[0:num_in_flight]]
>             del c_key_hist[:num_in_flight]
>             del c_all_players[0:num_in_flight]
>     day_1_flights()
>     
>     def day_2_to_5_flights():
>         flights = {}
>         for i in range(2,6):
>             print('...............day_' + str(i)+'................')
>             flights['a_flight_day_'+str(i)]= []
>             flights['b_flight_day_'+str(i)]= []
>             flights['c_flight_day_'+str(i)]= []
>             flights['d_flight_day_'+str(i)]= []            
>             lead = list('abcd')                   
>             flight_list = [flights['a_flight_day_'+str(i)],
> flights['b_flight_day_'+str(i)],\
>                            flights['c_flight_day_'+str(i)],
> flights['d_flight_day_'+str(i)]]
> 
>             for j in range(len(flight_list)):            
>                 def flight(cond, day):
>                     for player in all_players:
>                         if player not in cond:
>                             day.extend(player)
>                             cond.extend(history[player])
>                             history[lead[j]].extend(player)
>                     day.extend(lead[j])
>                     day.sort()
>                     [history[pl].extend(day) for pl in day[1:]]
>                     return lead[j]+'_flight_day_'+str(i)+ ': ' +
> str(flight_list[j])
>                    
>                 conditions = [history[lead[j]], history[lead[j]] +
> flights['a_flight_day_'+str(i)],\
>                               history[lead[j]] +
> flights['a_flight_day_'+str(i)] + \
>                               flights['b_flight_day_'+str(i)], \
>                               history[lead[j]] +
> flights['a_flight_day_'+str(i)] + \
>                               flights['b_flight_day_'+str(i)]+
> flights['c_flight_day_'+str(i)]] 
>                 print(flight(list(set(conditions[j])), flight_list[j]))
>     day_2_to_5_flights()
> startlist(list('abcdefghijklmnop'), 4)
> 
>  Many thanks, Marcus.
...

> The word "hardcoded" immediately stopped me in my tracks!
> 
> The whole point of using the computer is to find 'repetition' and have the
> machine/software save us from such boredom (or nit-picking detail in which
> we might make an error/become bored).
...

> The other 'side' of both of these code-constructs is the data-construct.
> Code-loops require data-collections! The hard-coded "a" and "day_1" made me
> shudder.
> (not a pretty sight - the code, nor me shuddering!)
...
> Sadly, the 'hard-coded' parts may 'help' sort-out week-one, but (IMHO) have
> made things impossibly-difficult to proceed into week-two (etc).
...


It works. Well done!
What could be better than that?



[Brutal] critique:

- ?pythonish? in German becomes ?pythonic? in English (but I'm sure we
all understood)

- position the two inner-functions outside and before startlist()

- whereas the ?4?, ie number of players per flight (num_in_flight), is
defined as a parameter in the call to startlist(), the five ?times or
days? is a 'magic constant' (worse, it appears in day_2_to_5_flights()
as part of ?range(2,6)? which 'disguises' it due to Python's way of working)

- the comments also include reference to those parameters as if they are
constants (which they are - if you only plan to use the algorithm for
this 16-4-5 configuration of the SGP). Thus, if the function were called
with different parameters, the comments would be not only wrong but have
the potential to mislead the reader

- in the same vein (as the two points above), the all_players (variable)
argument is followed by the generation of history as a list of constants
(?constant? cf ?variable?)

- on top of which: day_1_flights() generates key_hist from history even
though it already exists as all_players

- the Python facility for a 'dummy value' (that will never be used, or
perhaps only 'plugged-in' to 'make things happen') is _ (the
under-score/under-line character), ie

    for _ in c_key_hist:

- an alternative to using a meaningless 'placeholder' with no
computational-purpose, such as _ or dummy_i, is to choose an identifier
which aids readability, eg

    for each_flight in c_key_hist

- well done for noting that a list-comprehension could be used to
generate history/ies. Two thoughts:

  1 could the two for-loops be combined into a single nested
list-comprehension?

  2 does the reader's mind have to 'change gears' between reading the
outer for-loop as a 'traditional-loop' structure, and then the
inner-loop as a list-comprehension? ie would it be better to use the
same type of code-construct for both?

- both the code- and data-structures of day_1_flights() seem rather
tortured (and tortuous), and some are unused and therefore unnecessary.
Might it be possible to simplify, if the control-code commences with:

for first_player_index in range( 0, len( all_players ), num_in_flight ):
    print( first_player_index,
           all_players[ first_player_index:
                        first_player_index+num_in_flight
                      ]
         )

NB the print() is to make the methodology 'visible'.

- the docstring for day_1_flights() is only partly-correct. Isn't the
function also creating and building the history set?

- that being the case, should the initial set-definition be moved inside
the function?

- functions should not depend upon global values. How does the history
'pass' from one function to another - which is allied to the question:
how do the functions know about values such as _all_players and
num_in_flight? To make the functions self-contained and ?independent?,
these values should be passed-in as parameters/arguments and/or return-ed

- much of the above also applies to day_2_to_5_flights()

- chief concern with day_2_to_5_flights() is: what happens to
d_flight_day_N if there are fewer/more than four players per flight, or
what if there are fewer/more than 5 flights?

- the observation that the same players would always be the 'lead' of a
flight, is solid. Thus, could the lead-list be generated from a
provided-parameter, rather than stated as a constant? Could that
construct (also) have been used in the earlier function?

- we know (by definition) that flight() is an unnecessary set of
conditions to apply during day_1, but could it be used nonetheless? If
so, could day_1_flights() be 'folded into' day_2_to_5_flights() instead
of having separate functions?
(yes, I recall talking about the essential differences in an earlier
post - and perhaps I'm biased because this was how I structured the
draft-solution)


[More than] enough for now?
-- 
Regards,
=dn

From wlfraed at ix.netcom.com  Sat Jun 25 22:55:14 2022
From: wlfraed at ix.netcom.com (Dennis Lee Bieber)
Date: Sat, 25 Jun 2022 22:55:14 -0400
Subject: [Tutor] Event loop logic question
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <t8k80n$dqc$1@ciao.gmane.io> <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>
 <t9840s$ab4$1@ciao.gmane.io> <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com>
Message-ID: <pgdfbhtv18jddre94kh3nugo5b2nhkg7lv@4ax.com>

On Sun, 26 Jun 2022 10:06:25 +1000, Phil <phillor9 at gmail.com> declaimed the
following:

	WARNING: LONG HARANGUE FOLLOWS

>
>I was looking for an easy way to repeatedly read the keyboard and 
>pygamezero fills the bill. The problem that I initially had was that the 
>update function is called many times per second and sending control 
>commands from there caused the receiving hardware to be overwhelmed and 
>then to crash. The UI is a 20 * 20 square and servers no use at all, 
>it's just a requirement as far as I can tell, and so does not need 
>updating. All I need is to read the arrow keys and to be able to quit 
>the programme.
>

	In most GUI frameworks -- update would be called whenever the framework
has processed an event (some may have an ability to process a couple of
events and collect changes for one update call). Pygame Zero apparently
uses update() to render changes to the GUI buffer, and a separate draw()
operation to copy the buffer to the screen.

	There is no inherent means of rate-limiting a GUI framework. YOU will
have to write code to do rate limiting either within the event handlers, or
in the update operation. That likely means you need to track system elapsed
time for each event, issue an actual update to your device only if enough
elapsed time has passed and there has been no change in event type; OR you
issue a command when the event type changes.

	Looking for repeated key events can be troublesome. The OS handles the
keyboard repeat rate, and sends them (events) to whatever
application/framework is asking for them. Most likely, that means you might
receive a keydown/keyup event pair for each repetition. The only way to
confirm is to have a test application log (probably to a file, as it can
buffer entries rather than slowly updating a screen) each event it sees.

	Don't know if it applies but...
https://www.mattlayman.com/blog/2019/teach-kid-code-pygame-zero/
"""
The common pieces that you must handle yourself in Pygame are pre-wired in
Pygame Zero. This dramatically lowers the barrier to entry, at the cost of
flexibility. It?s absolutely the right trade-off for teaching.
"""
... could mean stuff you are trying to do is not really available.

	However, from the documentation, the best option for rate limiting is
to use	clock.schedule_interval()	configured to call a function that
issues commands to the end device, it will use an interval that will not
overload the device (you may have to experiment -- 1.0/20.0 would produce
20 commands per second).

	You will NOT issue commands from .update() or .draw() (though you might
use them to have that "unused" GUI window produce a heartbeat display).

	Your .on_key_down() and .on_key_up() callbacks just set a state flag
(I'm assuming more than binary state -- EXIT, IDLE, FORWARD, whatever else.
But you do need to verify that holding a key down doesn't result in
multiple down/up calls as the system repeats the key. If it does, you will
have more difficulties since the key handler state could go from, say
FORWARD to IDLE before the scheduler checks for command sending.

	I'm tempted to suggest that you ignore key_up events. You set the state
for the depressed key, and have the scheduler clear the state to "IDLE"
(but don't issue the idle command on this invocation). If no key has been
pressed since the scheduler cleared it, the next scheduled time will see
the IDLE and issue the proper command. Otherwise, a repeating key will be
another key_down event on return from the scheduler, and the event handler
will just reset the state to the proper value for that key.


	If the "GUI" is not really being used, why even invoke a GUI framework.
Just run a console application.

	On Windows, there is the msvcrt module in the standard library. It has
functions for 
"""
msvcrt.kbhit()
    Return True if a keypress is waiting to be read.

msvcrt.getch()
    Read a keypress and return the resulting character as a byte string.
Nothing is echoed to the console. This call will block if a keypress is not
already available, but will not wait for Enter to be pressed. If the
pressed key was a special function key, this will return '\000' or '\xe0';
the next call will return the keycode. The Control-C keypress cannot be
read with this function.
"""

	For Linux systems, there are a few modules to terminal IO control (but
understanding what you'd need to get characters as entered. Might be easier
to create a curses application (which should be available for Linux,
Windows ports are more problematic).
"""
curses.cbreak()
    Enter cbreak mode. In cbreak mode (sometimes called ?rare? mode) normal
tty line buffering is turned off and characters are available to be read
one by one. However, unlike raw mode, special characters (interrupt, quit,
suspend, and flow control) retain their effects on the tty driver and
calling program. Calling first raw() then cbreak() leaves the terminal in
cbreak mode.
"""
Note that doesn't mention key down/key up.
"""
curses.noecho()
    Leave echo mode. Echoing of input characters is turned off.
"""
"""
window.getch([y, x])
    Get a character. Note that the integer returned does not have to be in
ASCII range: function keys, keypad keys and so on are represented by
numbers higher than 255. In no-delay mode, return -1 if there is no input,
otherwise wait until a key is pressed.


window.getkey([y, x])
    Get a character, returning a string instead of an integer, as getch()
does. Function keys, keypad keys and other special keys return a multibyte
string containing the key name. In no-delay mode, raise an exception if
there is no input.
"""
Don't see an equivalent for .kbhit()


-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed at ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/


From phillor9 at gmail.com  Sat Jun 25 23:56:52 2022
From: phillor9 at gmail.com (Phil)
Date: Sun, 26 Jun 2022 13:56:52 +1000
Subject: [Tutor] Event loop logic question
In-Reply-To: <pgdfbhtv18jddre94kh3nugo5b2nhkg7lv@4ax.com>
References: <c10f88ca-fb65-ad13-9bd3-be4c8ab33f3e@gmail.com>
 <t8k80n$dqc$1@ciao.gmane.io> <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com>
 <t9840s$ab4$1@ciao.gmane.io> <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com>
 <pgdfbhtv18jddre94kh3nugo5b2nhkg7lv@4ax.com>
Message-ID: <e1683a3c-d54e-aacc-e45e-8d3ac4cd1265@gmail.com>


On 26/6/22 12:55, Dennis Lee Bieber wrote:
> On Sun, 26 Jun 2022 10:06:25 +1000, Phil <phillor9 at gmail.com> declaimed the
> following:
>
> 	WARNING: LONG HARANGUE FOLLOWS
>
> If the "GUI" is not really being used, why even invoke a GUI framework.
> Just run a console application.

Thank you Dennis,

I have used ncurses in the past, and it's no doubt the best option, but 
I couldn't remember how to read the keyboard but I do remember that it 
wasn't straight forward so I chose the simplest option. I've completed 
the project and about to move onto something else but I'll have another 
look at ncurses just as an exercise.

Maybe there's an alternative to ncurses for Linux, I'll check.

-- 

Regards,
Phil


From alexkleider at gmail.com  Sun Jun 26 15:37:43 2022
From: alexkleider at gmail.com (Alex Kleider)
Date: Sun, 26 Jun 2022 12:37:43 -0700
Subject: [Tutor] Feedback on coding style
In-Reply-To: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
Message-ID: <CAMCEyD43bLJ8GnKvpp9MqFXYVARnmj7F1M5bZOsPuy7A8i-6xw@mail.gmail.com>

I don't see this code on your github account.
(https://github.com/LeamHall?tab=repositories)
I'd be interested in seeing the current version having done something
somewhat related.
(https://github.com/alexKleider/blood-pressure-record)
My code needs a little updating (with regard to the README at least)
which I'll do if prodded a bit:-)
a
PS Disclaimer: python for me is an avocation, not a vocation!

On Mon, May 9, 2022 at 5:03 AM Leam Hall <leamhall at gmail.com> wrote:
>
> Hey all,
>
> I'm looking for general Python code critique, feel free to share snarky comments.  :)  The parameters are:
>   1. Code to Python 3.6 or so.
>   2. Use only the standard library or locally created modules.
>   3. Keep it clean and simple so new Pythonistas can understand and contribute.
>
> Let me know how to make this better.
>
> Leam
> --
> Automation Engineer        (reuel.net/resume)
> Scribe: The Domici War     (domiciwar.net)
> General Ne'er-do-well      (github.com/LeamHall)
>
> ###
>
> #!/usr/bin/env python3
>
> # name:     bp_tracker.py
> # version:  0.0.1
> # date:     20220509
> # author:   Leam Hall
> # desc:     Track and report on blood pressure numbers.
>
> # Notes:
> #  Datafile expects three ints and one float, in order.
>
> # TODO
> #   Add statistical analysis for standard deviation.
> #   Report based on time of day (early, midmorning, afternoon, evening)
> #   (?) Add current distance from goal?
>
> import argparse
> from datetime import datetime
> import os.path
>
> def array_from_file(report_file):
>    data = []
>    with open(report_file, 'r') as file:
>      for line in file:
>        line.strip()
>        datum = line.split()
>        if len(datum) == 4:
>          data.append(datum)
>        else:
>          continue
>    return data
>
> def report(report_data):
>    highest_systolic  = 0
>    highest_diastolic = 0
>    highest_pulse     = 0
>    latest            = -1.0
>    for datum in report_data:
>      systolic  = int(datum[0])
>      diastolic = int(datum[1])
>      pulse     = int(datum[2])
>      date      = float(datum[3])
>      if systolic > highest_systolic:
>        highest_systolic = systolic
>        highest_systolic_event = datum
>      if diastolic > highest_diastolic:
>        highest_diastolic = diastolic
>        highest_diastolic_event = datum
>      if pulse > highest_pulse:
>        highest_pulse = pulse
>        highest_pulse_event = datum
>      if date > latest:
>        latest_record = datum
>
>    print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event))
>    print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event))
>    print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event))
>    print("Latest Record: {}/{} {} {}".format(*latest_record))
>
> def result_string(report_list):
>    return "{} {} {} {}".format(*report_list)
>
> report_file = "bp_numbers.txt"
>
> parser = argparse.ArgumentParser()
> parser.add_argument("-a", "--add", nargs=3,
>    help = "Add in the order of systolic, diastolic, pulse")
> parser.add_argument("-f", "--file", help = "Report file")
> args    = parser.parse_args()
>
> if args.file:
>    report_file = args.file
>
> if args.add:
>    # This format allows sequencing now and parsing later.
>    timestamp   = datetime.now().strftime("%Y%m%d.%H%M")
>    this_report = args.add
>    this_report.append(timestamp)
>    with open(report_file, 'a') as file:
>      file.write(result_string(this_report) + "\n")
> else:
>    # Default behavior is to report.
>    if os.path.exists(report_file):
>      try:
>        report_data = array_from_file(report_file)
>        report(report_data)
>      except:
>        print("Error processing report data")
>    else:
>      print("Cannot find ", report_file)
>
> ###
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor



-- 
alex at kleider.ca  (sent from my current gizmo)

From leamhall at gmail.com  Sun Jun 26 15:53:32 2022
From: leamhall at gmail.com (Leam Hall)
Date: Sun, 26 Jun 2022 14:53:32 -0500
Subject: [Tutor] Feedback on coding style
In-Reply-To: <CAMCEyD43bLJ8GnKvpp9MqFXYVARnmj7F1M5bZOsPuy7A8i-6xw@mail.gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
 <CAMCEyD43bLJ8GnKvpp9MqFXYVARnmj7F1M5bZOsPuy7A8i-6xw@mail.gmail.com>
Message-ID: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com>

Hey Alex, here's the code:

	https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py

I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example:

	150 93 73 20220512.0832

That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format.

Leam


On 6/26/22 14:37, Alex Kleider wrote:
> I don't see this code on your github account.
> (https://github.com/LeamHall?tab=repositories)
> I'd be interested in seeing the current version having done something
> somewhat related.
> (https://github.com/alexKleider/blood-pressure-record)
> My code needs a little updating (with regard to the README at least)
> which I'll do if prodded a bit:-)
> a
> PS Disclaimer: python for me is an avocation, not a vocation!
> 
> On Mon, May 9, 2022 at 5:03 AM Leam Hall <leamhall at gmail.com> wrote:
>>
>> Hey all,
>>
>> I'm looking for general Python code critique, feel free to share snarky comments.  :)  The parameters are:
>>    1. Code to Python 3.6 or so.
>>    2. Use only the standard library or locally created modules.
>>    3. Keep it clean and simple so new Pythonistas can understand and contribute.
>>
>> Let me know how to make this better.
>>
>> Leam
>> --
>> Automation Engineer        (reuel.net/resume)
>> Scribe: The Domici War     (domiciwar.net)
>> General Ne'er-do-well      (github.com/LeamHall)
>>
>> ###
>>
>> #!/usr/bin/env python3
>>
>> # name:     bp_tracker.py
>> # version:  0.0.1
>> # date:     20220509
>> # author:   Leam Hall
>> # desc:     Track and report on blood pressure numbers.
>>
>> # Notes:
>> #  Datafile expects three ints and one float, in order.
>>
>> # TODO
>> #   Add statistical analysis for standard deviation.
>> #   Report based on time of day (early, midmorning, afternoon, evening)
>> #   (?) Add current distance from goal?
>>
>> import argparse
>> from datetime import datetime
>> import os.path
>>
>> def array_from_file(report_file):
>>     data = []
>>     with open(report_file, 'r') as file:
>>       for line in file:
>>         line.strip()
>>         datum = line.split()
>>         if len(datum) == 4:
>>           data.append(datum)
>>         else:
>>           continue
>>     return data
>>
>> def report(report_data):
>>     highest_systolic  = 0
>>     highest_diastolic = 0
>>     highest_pulse     = 0
>>     latest            = -1.0
>>     for datum in report_data:
>>       systolic  = int(datum[0])
>>       diastolic = int(datum[1])
>>       pulse     = int(datum[2])
>>       date      = float(datum[3])
>>       if systolic > highest_systolic:
>>         highest_systolic = systolic
>>         highest_systolic_event = datum
>>       if diastolic > highest_diastolic:
>>         highest_diastolic = diastolic
>>         highest_diastolic_event = datum
>>       if pulse > highest_pulse:
>>         highest_pulse = pulse
>>         highest_pulse_event = datum
>>       if date > latest:
>>         latest_record = datum
>>
>>     print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event))
>>     print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event))
>>     print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event))
>>     print("Latest Record: {}/{} {} {}".format(*latest_record))
>>
>> def result_string(report_list):
>>     return "{} {} {} {}".format(*report_list)
>>
>> report_file = "bp_numbers.txt"
>>
>> parser = argparse.ArgumentParser()
>> parser.add_argument("-a", "--add", nargs=3,
>>     help = "Add in the order of systolic, diastolic, pulse")
>> parser.add_argument("-f", "--file", help = "Report file")
>> args    = parser.parse_args()
>>
>> if args.file:
>>     report_file = args.file
>>
>> if args.add:
>>     # This format allows sequencing now and parsing later.
>>     timestamp   = datetime.now().strftime("%Y%m%d.%H%M")
>>     this_report = args.add
>>     this_report.append(timestamp)
>>     with open(report_file, 'a') as file:
>>       file.write(result_string(this_report) + "\n")
>> else:
>>     # Default behavior is to report.
>>     if os.path.exists(report_file):
>>       try:
>>         report_data = array_from_file(report_file)
>>         report(report_data)
>>       except:
>>         print("Error processing report data")
>>     else:
>>       print("Cannot find ", report_file)
>>
>> ###
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
> 
> 
> 

-- 
Automation Engineer        (reuel.net/resume)
Scribe: The Domici War     (domiciwar.net)
General Ne'er-do-well      (github.com/LeamHall)

From alexkleider at gmail.com  Sun Jun 26 16:10:26 2022
From: alexkleider at gmail.com (Alex Kleider)
Date: Sun, 26 Jun 2022 13:10:26 -0700
Subject: [Tutor] Feedback on coding style
In-Reply-To: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
 <CAMCEyD43bLJ8GnKvpp9MqFXYVARnmj7F1M5bZOsPuy7A8i-6xw@mail.gmail.com>
 <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com>
Message-ID: <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>

Thanks for (so promptly!) getting back to me.
I already tried that and was unsuccessful:
(perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/
(perci) alex at t460:~/Git/LH$ git clone
https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
Cloning into 'bp_tracker.py'...
fatal: repository
'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/'
not found
(perci) alex at t460:~/Git/LH$

I was able (successfully) to clone your resume_writer:
$ git clone https://github.com/LeamHall/resume_writer
I'm not very proficient with git and github (which is probably pretty
apparent:-)
The bp_tracker seems to be embedded within something else so I'm
guessing that's what's leading to the error.
a

On Sun, Jun 26, 2022 at 12:55 PM Leam Hall <leamhall at gmail.com> wrote:
>
> Hey Alex, here's the code:
>
>         https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
>
> I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example:
>
>         150 93 73 20220512.0832
>
> That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format.
>
> Leam
>
>
> On 6/26/22 14:37, Alex Kleider wrote:
> > I don't see this code on your github account.
> > (https://github.com/LeamHall?tab=repositories)
> > I'd be interested in seeing the current version having done something
> > somewhat related.
> > (https://github.com/alexKleider/blood-pressure-record)
> > My code needs a little updating (with regard to the README at least)
> > which I'll do if prodded a bit:-)
> > a
> > PS Disclaimer: python for me is an avocation, not a vocation!
> >
> > On Mon, May 9, 2022 at 5:03 AM Leam Hall <leamhall at gmail.com> wrote:
> >>
> >> Hey all,
> >>
> >> I'm looking for general Python code critique, feel free to share snarky comments.  :)  The parameters are:
> >>    1. Code to Python 3.6 or so.
> >>    2. Use only the standard library or locally created modules.
> >>    3. Keep it clean and simple so new Pythonistas can understand and contribute.
> >>
> >> Let me know how to make this better.
> >>
> >> Leam
> >> --
> >> Automation Engineer        (reuel.net/resume)
> >> Scribe: The Domici War     (domiciwar.net)
> >> General Ne'er-do-well      (github.com/LeamHall)
> >>
> >> ###
> >>
> >> #!/usr/bin/env python3
> >>
> >> # name:     bp_tracker.py
> >> # version:  0.0.1
> >> # date:     20220509
> >> # author:   Leam Hall
> >> # desc:     Track and report on blood pressure numbers.
> >>
> >> # Notes:
> >> #  Datafile expects three ints and one float, in order.
> >>
> >> # TODO
> >> #   Add statistical analysis for standard deviation.
> >> #   Report based on time of day (early, midmorning, afternoon, evening)
> >> #   (?) Add current distance from goal?
> >>
> >> import argparse
> >> from datetime import datetime
> >> import os.path
> >>
> >> def array_from_file(report_file):
> >>     data = []
> >>     with open(report_file, 'r') as file:
> >>       for line in file:
> >>         line.strip()
> >>         datum = line.split()
> >>         if len(datum) == 4:
> >>           data.append(datum)
> >>         else:
> >>           continue
> >>     return data
> >>
> >> def report(report_data):
> >>     highest_systolic  = 0
> >>     highest_diastolic = 0
> >>     highest_pulse     = 0
> >>     latest            = -1.0
> >>     for datum in report_data:
> >>       systolic  = int(datum[0])
> >>       diastolic = int(datum[1])
> >>       pulse     = int(datum[2])
> >>       date      = float(datum[3])
> >>       if systolic > highest_systolic:
> >>         highest_systolic = systolic
> >>         highest_systolic_event = datum
> >>       if diastolic > highest_diastolic:
> >>         highest_diastolic = diastolic
> >>         highest_diastolic_event = datum
> >>       if pulse > highest_pulse:
> >>         highest_pulse = pulse
> >>         highest_pulse_event = datum
> >>       if date > latest:
> >>         latest_record = datum
> >>
> >>     print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event))
> >>     print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event))
> >>     print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event))
> >>     print("Latest Record: {}/{} {} {}".format(*latest_record))
> >>
> >> def result_string(report_list):
> >>     return "{} {} {} {}".format(*report_list)
> >>
> >> report_file = "bp_numbers.txt"
> >>
> >> parser = argparse.ArgumentParser()
> >> parser.add_argument("-a", "--add", nargs=3,
> >>     help = "Add in the order of systolic, diastolic, pulse")
> >> parser.add_argument("-f", "--file", help = "Report file")
> >> args    = parser.parse_args()
> >>
> >> if args.file:
> >>     report_file = args.file
> >>
> >> if args.add:
> >>     # This format allows sequencing now and parsing later.
> >>     timestamp   = datetime.now().strftime("%Y%m%d.%H%M")
> >>     this_report = args.add
> >>     this_report.append(timestamp)
> >>     with open(report_file, 'a') as file:
> >>       file.write(result_string(this_report) + "\n")
> >> else:
> >>     # Default behavior is to report.
> >>     if os.path.exists(report_file):
> >>       try:
> >>         report_data = array_from_file(report_file)
> >>         report(report_data)
> >>       except:
> >>         print("Error processing report data")
> >>     else:
> >>       print("Cannot find ", report_file)
> >>
> >> ###
> >> _______________________________________________
> >> Tutor maillist  -  Tutor at python.org
> >> To unsubscribe or change subscription options:
> >> https://mail.python.org/mailman/listinfo/tutor
> >
> >
> >
>
> --
> Automation Engineer        (reuel.net/resume)
> Scribe: The Domici War     (domiciwar.net)
> General Ne'er-do-well      (github.com/LeamHall)
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor



-- 
alex at kleider.ca  (sent from my current gizmo)

From leamhall at gmail.com  Sun Jun 26 16:21:43 2022
From: leamhall at gmail.com (Leam Hall)
Date: Sun, 26 Jun 2022 15:21:43 -0500
Subject: [Tutor] Feedback on coding style
In-Reply-To: <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
 <CAMCEyD43bLJ8GnKvpp9MqFXYVARnmj7F1M5bZOsPuy7A8i-6xw@mail.gmail.com>
 <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com>
 <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>
Message-ID: <54cf42fc-bbed-63f0-e096-0e3481b14def@gmail.com>

Happy to help!

My apologies, I sent you the direct link to the code. You should be able to clone:

	https://github.com/LeamHall/admin_tools

and then it'll be in there. You need to create the "data" directory, and then run it with an "add" option to create the file. Of course, if you don't care about the timestamps at first, you can just copy in the data, where each line is:

	systolic diastolic pulse 0

The last 0 is the timestamp position, and each element in the line is white space delimited. you can manually enter the timestamp, if you like.

Leam
   

On 6/26/22 15:10, Alex Kleider wrote:
> Thanks for (so promptly!) getting back to me.
> I already tried that and was unsuccessful:
> (perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/
> (perci) alex at t460:~/Git/LH$ git clone
> https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
> Cloning into 'bp_tracker.py'...
> fatal: repository
> 'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/'
> not found
> (perci) alex at t460:~/Git/LH$
> 
> I was able (successfully) to clone your resume_writer:
> $ git clone https://github.com/LeamHall/resume_writer
> I'm not very proficient with git and github (which is probably pretty
> apparent:-)
> The bp_tracker seems to be embedded within something else so I'm
> guessing that's what's leading to the error.
> a
> 
> On Sun, Jun 26, 2022 at 12:55 PM Leam Hall <leamhall at gmail.com> wrote:
>>
>> Hey Alex, here's the code:
>>
>>          https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
>>
>> I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example:
>>
>>          150 93 73 20220512.0832
>>
>> That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format.
>>
>> Leam
>>
>>
>> On 6/26/22 14:37, Alex Kleider wrote:
>>> I don't see this code on your github account.
>>> (https://github.com/LeamHall?tab=repositories)
>>> I'd be interested in seeing the current version having done something
>>> somewhat related.
>>> (https://github.com/alexKleider/blood-pressure-record)
>>> My code needs a little updating (with regard to the README at least)
>>> which I'll do if prodded a bit:-)
>>> a
>>> PS Disclaimer: python for me is an avocation, not a vocation!
>>>
>>> On Mon, May 9, 2022 at 5:03 AM Leam Hall <leamhall at gmail.com> wrote:
>>>>
>>>> Hey all,
>>>>
>>>> I'm looking for general Python code critique, feel free to share snarky comments.  :)  The parameters are:
>>>>     1. Code to Python 3.6 or so.
>>>>     2. Use only the standard library or locally created modules.
>>>>     3. Keep it clean and simple so new Pythonistas can understand and contribute.
>>>>
>>>> Let me know how to make this better.
>>>>
>>>> Leam
>>>> --
>>>> Automation Engineer        (reuel.net/resume)
>>>> Scribe: The Domici War     (domiciwar.net)
>>>> General Ne'er-do-well      (github.com/LeamHall)
>>>>
>>>> ###
>>>>
>>>> #!/usr/bin/env python3
>>>>
>>>> # name:     bp_tracker.py
>>>> # version:  0.0.1
>>>> # date:     20220509
>>>> # author:   Leam Hall
>>>> # desc:     Track and report on blood pressure numbers.
>>>>
>>>> # Notes:
>>>> #  Datafile expects three ints and one float, in order.
>>>>
>>>> # TODO
>>>> #   Add statistical analysis for standard deviation.
>>>> #   Report based on time of day (early, midmorning, afternoon, evening)
>>>> #   (?) Add current distance from goal?
>>>>
>>>> import argparse
>>>> from datetime import datetime
>>>> import os.path
>>>>
>>>> def array_from_file(report_file):
>>>>      data = []
>>>>      with open(report_file, 'r') as file:
>>>>        for line in file:
>>>>          line.strip()
>>>>          datum = line.split()
>>>>          if len(datum) == 4:
>>>>            data.append(datum)
>>>>          else:
>>>>            continue
>>>>      return data
>>>>
>>>> def report(report_data):
>>>>      highest_systolic  = 0
>>>>      highest_diastolic = 0
>>>>      highest_pulse     = 0
>>>>      latest            = -1.0
>>>>      for datum in report_data:
>>>>        systolic  = int(datum[0])
>>>>        diastolic = int(datum[1])
>>>>        pulse     = int(datum[2])
>>>>        date      = float(datum[3])
>>>>        if systolic > highest_systolic:
>>>>          highest_systolic = systolic
>>>>          highest_systolic_event = datum
>>>>        if diastolic > highest_diastolic:
>>>>          highest_diastolic = diastolic
>>>>          highest_diastolic_event = datum
>>>>        if pulse > highest_pulse:
>>>>          highest_pulse = pulse
>>>>          highest_pulse_event = datum
>>>>        if date > latest:
>>>>          latest_record = datum
>>>>
>>>>      print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event))
>>>>      print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event))
>>>>      print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event))
>>>>      print("Latest Record: {}/{} {} {}".format(*latest_record))
>>>>
>>>> def result_string(report_list):
>>>>      return "{} {} {} {}".format(*report_list)
>>>>
>>>> report_file = "bp_numbers.txt"
>>>>
>>>> parser = argparse.ArgumentParser()
>>>> parser.add_argument("-a", "--add", nargs=3,
>>>>      help = "Add in the order of systolic, diastolic, pulse")
>>>> parser.add_argument("-f", "--file", help = "Report file")
>>>> args    = parser.parse_args()
>>>>
>>>> if args.file:
>>>>      report_file = args.file
>>>>
>>>> if args.add:
>>>>      # This format allows sequencing now and parsing later.
>>>>      timestamp   = datetime.now().strftime("%Y%m%d.%H%M")
>>>>      this_report = args.add
>>>>      this_report.append(timestamp)
>>>>      with open(report_file, 'a') as file:
>>>>        file.write(result_string(this_report) + "\n")
>>>> else:
>>>>      # Default behavior is to report.
>>>>      if os.path.exists(report_file):
>>>>        try:
>>>>          report_data = array_from_file(report_file)
>>>>          report(report_data)
>>>>        except:
>>>>          print("Error processing report data")
>>>>      else:
>>>>        print("Cannot find ", report_file)
>>>>
>>>> ###
>>>> _______________________________________________
>>>> Tutor maillist  -  Tutor at python.org
>>>> To unsubscribe or change subscription options:
>>>> https://mail.python.org/mailman/listinfo/tutor
>>>
>>>
>>>
>>
>> --
>> Automation Engineer        (reuel.net/resume)
>> Scribe: The Domici War     (domiciwar.net)
>> General Ne'er-do-well      (github.com/LeamHall)
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
> 
> 
> 

-- 
Automation Engineer        (reuel.net/resume)
Scribe: The Domici War     (domiciwar.net)
General Ne'er-do-well      (github.com/LeamHall)

From PythonList at DancesWithMice.info  Sun Jun 26 19:10:13 2022
From: PythonList at DancesWithMice.info (dn)
Date: Mon, 27 Jun 2022 11:10:13 +1200
Subject: [Tutor] Feedback on coding style
In-Reply-To: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
Message-ID: <b1f9b80c-7843-b0a1-f324-34bb9938156c@DancesWithMice.info>

On 10/05/2022 00.01, Leam Hall wrote:
> Hey all,
> 
> I'm looking for general Python code critique, feel free to share snarky
> comments.? :)? The parameters are:
> ?1. Code to Python 3.6 or so.
> ?2. Use only the standard library or locally created modules.
> ?3. Keep it clean and simple so new Pythonistas can understand and
> contribute.
> 
> Let me know how to make this better.


Apologies: was 'otherwise occupied' when this first hit the list. (and
you're not Swiss, German, or ... so will try to stay away from what may
seem like the 'snarky' way to reply)


- as a reader, will I be more-interested in the location of your PC's
python3 interpreter, the program[me]'s name, its version, origin-date,
author, or its desc[ription]?

- the university requires me to establish 'ownership' of all materials
in case they are (later) used in courseware, articles, books, etc. Thus
a rather legalistic line! Here is the standard 'base' used in a typical
toy-example used to illustrate a particular concept during a local
PUG-meeting:

<<<
""" Generator expression. """

__author__ = "David Neil, IT&T Consultant"
__copyright__ = "Copyright ? 2022~"
__license__ = "May not be copied"
__python__ = "3.9"
>>>

Thus, an alternative to earlier suggestions of 'bundling' everything
into a single docstring: the module docstring is separate, and each of
these items is retrievable. Herewith:

>>> import tuple_comprehension_demo as tcd
>>> help( tcd )

Help on module tuple_comprehension_demo:

NAME
    tuple_comprehension_demo - Generator expression.

DATA
    __copyright__ = 'Copyright ? 2022~'
    __license__ = 'May not be copied'
    __python__ = '3.9'
    student_names = ('fred', 'wiLma', 'bARNEy', 'bettY', 'Pete', 'dn')
    tuple_of_names = ('Fred', 'Wilma', 'Barney', 'Betty', 'Pete', 'Dn')

AUTHOR
    David Neil, IT&T Consultant

FILE
    /home/dn/Projects/AuckPUG-2022-04-20/tuple_comprehension_demo.py
>>>
PS I'll be interested to see what 'normal people' do for this...

- emphasising @Alan's comments: we (old-timers - me, not Alan!) used to
talk about Input-Process-Output, ie in order to produce the output
required, what processing will need to be done, and thus: what input(s)
will be needed. I suggest that this is still a valid approach to
decomposing a problem, and applies particularly well to the (two modes)
way this code is to work - in fact, consider adding a fourth 'phase':
'Establish the Environment' (all the argparse stuff which will define
which mode the user wishes to operate). Accordingly, the four 'phases'
guide us into a 'first-layer' thinking-about/designing, and then coding
solutions. More importantly (especially given the discussions about
flat-file, CSV, and SQL storage options), if the code is split into
*independent* sections, it becomes easy to change (or experiment with
changes/upgrades) because only the one section will be affected - and
the rest of the code should continue to operate (correctly, untouched,
in perfect splendor...)

- just because there was talk about options for persistence, I'll
throw-in the suggestion of MongoDB (or similar). NoSQL DBs are a lot
simpler to use than SQL (and I learned SQL before Oracle or SQL/DS (IBM
DB2) were even commercial products, and train in the subject to this
day!). More to the point, their data appears as if it was a Python
data-structure (a list in this case), so no need to bend-your-head
around much that is 'new'! (similar argument applies to JSON, YAML, etc)

- for SCM (source code management) there are several good tools (I don't
do CI/CD). I use git on my local machines, and for security-copies and
sharing, push and pull to my GitLab or others'. One of the perqs of
joining the local Open Source Society is access to a number of servers
and services. One of which is a GitLab instance. That Microsoft cannot
claim 'ownership' or otherwise use/make use of 'my' code (see legalism,
above) is an important legal, if not, moral component to my decision (or
that imposed upon me). GitLab.com tells us how much better (more mature)
their offering is than GitHub (perhaps 'the other guys' do the same, to
their benefit?), and outlines their $free/$paid service plans.

- should you move to SCM, then suggest review of the header-comments,
because dates and version-numbers become a function of the SCM rather
than a manual coding-process!

- recommend researching named-tuples
(https://docs.python.org/3/library/collections.html?highlight=namedtuple#collections.namedtuple)
then the datum can be given a less-generic name, eg
blood_pressure_reading; and the three components similarly identified.
Because each element of the tuple can be given its own name (hence the
"named", hah!) there is no need for packing/unpacking between int(s) and
the collection data-structure!

- reiterating: 'names' are important. (yes, this is a short and simple
module, but habits are hard to form - and all-to easy to slip) Thus,
generic terms such as "report", "result_string", and "report_list" seem
 obvious at the time of coding - but will require more
comprehension-effort (than necessary) at some (six-months in the) future
time-of-reading! With today's text-editors able to pop-up suggestions
for what we might be intending to type, identifier length (and thus
precision-naming) is hardly an issue (cf temptations towards
laziness/saving my worn-down fingers, ...)

- unlabelled lists of 'anything' decrease readability! Accordingly, when
it comes to function declarations (obvious example), after about two
parameters (IMHO) we should be using keyword-arguments. The problem with
format() (particularly in this example-code) is that it is preceded by a
bunch of empty braces, and these have to be positionally-related to some
code 'further over on the right' (of the code-line). Admittedly in this
case things are not too bad because tuple-unpacking is in-use - so we
don't have to keep track of four pairs of braces (on the left), and four
identifiers (on the right). [it's a lot of effort for aged eyes!]
Recommend researching f-strings
(https://docs.python.org/3/reference/lexical_analysis.html#f-strings).
The process becomes an expression (cf a function) but doesn't take much
learning because the same format() protocol is applied.

- finally, if you can persuade my at-home BP-machine to 'talk' to my
computer, sign me up! At this time, I would have to copy from my
bed-side scratch-pad into the PC, manually. (am too lazy for that!)
Which does lead to the observation that now() is only applicable if the
input is carried-out immediately after the BP-test - which saves adding
another input field, but there's a human involved...
-- 
Regards,
=dn

From leamhall at gmail.com  Sun Jun 26 20:12:19 2022
From: leamhall at gmail.com (Leam Hall)
Date: Sun, 26 Jun 2022 19:12:19 -0500
Subject: [Tutor] Feedback on coding style
In-Reply-To: <b1f9b80c-7843-b0a1-f324-34bb9938156c@DancesWithMice.info>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
 <b1f9b80c-7843-b0a1-f324-34bb9938156c@DancesWithMice.info>
Message-ID: <ac709a8a-7ded-7001-28f5-8a7624545d45@gmail.com>

No apologies needed! Your input is habitually useful and welcome.

It has taken me years, literally (I'm a bit slow), to figure out my coding style. While Python is adept in a lot of markets, I'm more of a tools person than an application programmer. My best work is usually to write something, make sure it's documented, train the gaining team on how it works, and then hand it off. In that environment, third-party libraries, venvs, and external application overhead (like MongoDB) don't really fit well. Plan text rules, and there's little reason for complex features.

For the record, I passed one of the MongoDB U Python developer's classes. Fun stuff, and I've been considering doing more. Sadly, MongoDB v5 needs a CPU flag that my old Dell doesn't have. So I'm on the obsolescence track already.

Of course, us tools folks are often tasked to write in different languages. The "#" symbol is a common, if not universal comment designator, and that's why I use it vice language specific things like "__author__". Keeping the interpreter and those lines up top really makes life easy when I have to sort through a few thousand files to track names and the last time it had major modifications.

Some years ago my brother, a better programmer than I, told me about "input-output-process" and I try to stick to that. In this case, the input is technically just three items, BP, pulse, and timestamp. Usually we talk about the two BP numbers as a single unit, and the only reason they are separate here is to make parsing easier. The pulse is almost ancillary, and the timestamp is for sorting if I ever get to "last X number of days", or maybe chart across morning, midday, and evening. Because the data is stored in a text file, I can edit (vim, thank you. None of that pretty stuff) the timestamp if it needs changing.

I'm on the fence about f-strings. Ruby has been doing that for years, and I usually prefer something that's common-ish between languages. Using printf or format() is less visually noisy, for me, than f-strings. f-strings have added a lot of usefulness, like Ruby, but much of what I do can be done before the print statement.

Does that make sense?

Leam


On 6/26/22 18:10, dn wrote:
> On 10/05/2022 00.01, Leam Hall wrote:
>> Hey all,
>>
>> I'm looking for general Python code critique, feel free to share snarky
>> comments.? :)? The parameters are:
>>  ?1. Code to Python 3.6 or so.
>>  ?2. Use only the standard library or locally created modules.
>>  ?3. Keep it clean and simple so new Pythonistas can understand and
>> contribute.
>>
>> Let me know how to make this better.
> 
> 
> Apologies: was 'otherwise occupied' when this first hit the list. (and
> you're not Swiss, German, or ... so will try to stay away from what may
> seem like the 'snarky' way to reply)
> 
> 
> - as a reader, will I be more-interested in the location of your PC's
> python3 interpreter, the program[me]'s name, its version, origin-date,
> author, or its desc[ription]?
> 
> - the university requires me to establish 'ownership' of all materials
> in case they are (later) used in courseware, articles, books, etc. Thus
> a rather legalistic line! Here is the standard 'base' used in a typical
> toy-example used to illustrate a particular concept during a local
> PUG-meeting:
> 
> <<<
> """ Generator expression. """
> 
> __author__ = "David Neil, IT&T Consultant"
> __copyright__ = "Copyright ? 2022~"
> __license__ = "May not be copied"
> __python__ = "3.9"
>>>>
> 
> Thus, an alternative to earlier suggestions of 'bundling' everything
> into a single docstring: the module docstring is separate, and each of
> these items is retrievable. Herewith:
> 
>>>> import tuple_comprehension_demo as tcd
>>>> help( tcd )
> 
> Help on module tuple_comprehension_demo:
> 
> NAME
>      tuple_comprehension_demo - Generator expression.
> 
> DATA
>      __copyright__ = 'Copyright ? 2022~'
>      __license__ = 'May not be copied'
>      __python__ = '3.9'
>      student_names = ('fred', 'wiLma', 'bARNEy', 'bettY', 'Pete', 'dn')
>      tuple_of_names = ('Fred', 'Wilma', 'Barney', 'Betty', 'Pete', 'Dn')
> 
> AUTHOR
>      David Neil, IT&T Consultant
> 
> FILE
>      /home/dn/Projects/AuckPUG-2022-04-20/tuple_comprehension_demo.py
>>>>
> PS I'll be interested to see what 'normal people' do for this...
> 
> - emphasising @Alan's comments: we (old-timers - me, not Alan!) used to
> talk about Input-Process-Output, ie in order to produce the output
> required, what processing will need to be done, and thus: what input(s)
> will be needed. I suggest that this is still a valid approach to
> decomposing a problem, and applies particularly well to the (two modes)
> way this code is to work - in fact, consider adding a fourth 'phase':
> 'Establish the Environment' (all the argparse stuff which will define
> which mode the user wishes to operate). Accordingly, the four 'phases'
> guide us into a 'first-layer' thinking-about/designing, and then coding
> solutions. More importantly (especially given the discussions about
> flat-file, CSV, and SQL storage options), if the code is split into
> *independent* sections, it becomes easy to change (or experiment with
> changes/upgrades) because only the one section will be affected - and
> the rest of the code should continue to operate (correctly, untouched,
> in perfect splendor...)
> 
> - just because there was talk about options for persistence, I'll
> throw-in the suggestion of MongoDB (or similar). NoSQL DBs are a lot
> simpler to use than SQL (and I learned SQL before Oracle or SQL/DS (IBM
> DB2) were even commercial products, and train in the subject to this
> day!). More to the point, their data appears as if it was a Python
> data-structure (a list in this case), so no need to bend-your-head
> around much that is 'new'! (similar argument applies to JSON, YAML, etc)
> 
> - for SCM (source code management) there are several good tools (I don't
> do CI/CD). I use git on my local machines, and for security-copies and
> sharing, push and pull to my GitLab or others'. One of the perqs of
> joining the local Open Source Society is access to a number of servers
> and services. One of which is a GitLab instance. That Microsoft cannot
> claim 'ownership' or otherwise use/make use of 'my' code (see legalism,
> above) is an important legal, if not, moral component to my decision (or
> that imposed upon me). GitLab.com tells us how much better (more mature)
> their offering is than GitHub (perhaps 'the other guys' do the same, to
> their benefit?), and outlines their $free/$paid service plans.
> 
> - should you move to SCM, then suggest review of the header-comments,
> because dates and version-numbers become a function of the SCM rather
> than a manual coding-process!
> 
> - recommend researching named-tuples
> (https://docs.python.org/3/library/collections.html?highlight=namedtuple#collections.namedtuple)
> then the datum can be given a less-generic name, eg
> blood_pressure_reading; and the three components similarly identified.
> Because each element of the tuple can be given its own name (hence the
> "named", hah!) there is no need for packing/unpacking between int(s) and
> the collection data-structure!
> 
> - reiterating: 'names' are important. (yes, this is a short and simple
> module, but habits are hard to form - and all-to easy to slip) Thus,
> generic terms such as "report", "result_string", and "report_list" seem
>   obvious at the time of coding - but will require more
> comprehension-effort (than necessary) at some (six-months in the) future
> time-of-reading! With today's text-editors able to pop-up suggestions
> for what we might be intending to type, identifier length (and thus
> precision-naming) is hardly an issue (cf temptations towards
> laziness/saving my worn-down fingers, ...)
> 
> - unlabelled lists of 'anything' decrease readability! Accordingly, when
> it comes to function declarations (obvious example), after about two
> parameters (IMHO) we should be using keyword-arguments. The problem with
> format() (particularly in this example-code) is that it is preceded by a
> bunch of empty braces, and these have to be positionally-related to some
> code 'further over on the right' (of the code-line). Admittedly in this
> case things are not too bad because tuple-unpacking is in-use - so we
> don't have to keep track of four pairs of braces (on the left), and four
> identifiers (on the right). [it's a lot of effort for aged eyes!]
> Recommend researching f-strings
> (https://docs.python.org/3/reference/lexical_analysis.html#f-strings).
> The process becomes an expression (cf a function) but doesn't take much
> learning because the same format() protocol is applied.
> 
> - finally, if you can persuade my at-home BP-machine to 'talk' to my
> computer, sign me up! At this time, I would have to copy from my
> bed-side scratch-pad into the PC, manually. (am too lazy for that!)
> Which does lead to the observation that now() is only applicable if the
> input is carried-out immediately after the BP-test - which saves adding
> another input field, but there's a human involved...

-- 
Automation Engineer        (reuel.net/resume)
Scribe: The Domici War     (domiciwar.net)
General Ne'er-do-well      (github.com/LeamHall)

From cs at cskk.id.au  Sun Jun 26 18:40:14 2022
From: cs at cskk.id.au (Cameron Simpson)
Date: Mon, 27 Jun 2022 08:40:14 +1000
Subject: [Tutor] Feedback on coding style
In-Reply-To: <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>
References: <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>
Message-ID: <YrjgTv8fuqUfps1T@cskk.homeip.net>

On 26Jun2022 13:10, Alex Kleider <alexkleider at gmail.com> wrote:
>Thanks for (so promptly!) getting back to me.
>I already tried that and was unsuccessful:
>(perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/
>(perci) alex at t460:~/Git/LH$ git clone
>https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
>Cloning into 'bp_tracker.py'...
>fatal: repository
>'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/'
>not found
>(perci) alex at t460:~/Git/LH$

bp_tracker.py is not a repository, so it cannot be cloned.
You can clone https://github.com/LeamHall/admin_tools instead.
Or go to 
https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py and 
use the download link on that page.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From PythonList at DancesWithMice.info  Sun Jun 26 21:02:11 2022
From: PythonList at DancesWithMice.info (dn)
Date: Mon, 27 Jun 2022 13:02:11 +1200
Subject: [Tutor] Feedback on coding style
In-Reply-To: <ac709a8a-7ded-7001-28f5-8a7624545d45@gmail.com>
References: <e4d22e97-e341-ace7-722e-ce33cfeb46f1@gmail.com>
 <b1f9b80c-7843-b0a1-f324-34bb9938156c@DancesWithMice.info>
 <ac709a8a-7ded-7001-28f5-8a7624545d45@gmail.com>
Message-ID: <9082cb7b-83e2-b6b9-a8ce-ccc39b881352@DancesWithMice.info>

On 27/06/2022 12.12, Leam Hall wrote:
> No apologies needed! Your input is habitually useful and welcome.
> 
> It has taken me years, literally (I'm a bit slow), to figure out my

This is not at all unusual. The phenomenon experienced (largely
unknowingly) by most of us, is that we start in a course (which only
requires short coding examples, and it is only sometime later that we
realise the virtue of some of this 'other stuff', like "style", after
we've some miles under our belts and the size of our tasks has increased
(markedly), 'complexity' has raised its ugly head...
(see also 'structure', 'objects', 'testing', ...)


> coding style. While Python is adept in a lot of markets, I'm more of a
> tools person than an application programmer. My best work is usually to
> write something, make sure it's documented, train the gaining team on
> how it works, and then hand it off. In that environment, third-party
> libraries, venvs, and external application overhead (like MongoDB) don't
> really fit well. Plan text rules, and there's little reason for complex
> features.

IMHO the opposite applies! The tools I build are used by trainers and
educators. There is no such thing as 'one size fits all', and we are
constantly innovating. A tool is discarded/replaced before anyone says
"it's done"!

Accordingly, a need to keep things tidy and 'professional'. Admittedly
one reason is that someone else might pick-up the tool and code the
extension - so the more 'help' they have, the better our team will perform.
(see also earlier wisdom (not mine) about "habits"...)

This is also a very good argument for SCM. Whilst it might seem like an
"overhead" (and yes, there is bound to be some learning to do), it adds
extra layers of flexibility (as well as coping with version-numbering
for you). For example, your first version involved a flat-file; then
someone else came-along and suggested SQLite. What if you had delivered
the first version to some 'client' and (s)he was perfectly happy -
perhaps doesn't have SQLite on his/her PC (as if). Now you have two
distinct versions to track. Just as you thought it was safe to get back
into the water, I come along with MongoDB. Then there were three!

SCMs offer features such as "branches" which will help to manage this.
Building upon the earlier 'structure' conversation, the SCM will also
help when upgrading some 'process' code, and creates a situation where
that can be "shipped" to every user, of every 'version' (because the
'process' code has been structured to be "independent" of the
storage-method!). This sort of thing requires all-manner of
mental-gymnastics to perform manually (or more likely, certainly in my
experience of this business, 'we' will keep-quiet and the 'other users'
will simply miss-out...).

The other handy side-effect is that documentation associated with
"branches" will enable you to keep-track of which users/groups use which
branch, and when they come to you with a problem/proposal (as they
inevitably will), you 'know' (can reproduce) their exact system, ie what
they think of as "the system"!

I guess it all depends upon one's experience of "hand it off"?

Yes, "features" should be countered by YAGNI. However, much of what was
discussed has the ultimate aim of making life easier for me, oh wait a
minute: it's for you!

Something I find helpful, you may not. Fair enough - there's that other
'principle': YMMV!


> For the record, I passed one of the MongoDB U Python developer's
> classes. Fun stuff, and I've been considering doing more. Sadly, MongoDB
> v5 needs a CPU flag that my old Dell doesn't have. So I'm on the
> obsolescence track already.

Warning: this comment may contain heresy!

Who says that we have to use the latest version? (can you hear the
collective sucking-in of breath? A little frisson of excitement?)


Often we don't use the latest version until 'they' have had time to make
sure it works properly, and then, only after we've gone through and
checked our various 'versions' of things to make sure that our stuff
still works as-advertised.
(it's not just 'our code' that should be 'tested'!)

An alternative (which may be in the "it's OK for you" category*) is to
use a cloud service. The likes of AWS, IBM, Oracle, etc, usually offer a
'free tier' and offer a range of RDBMS or NoSQL options.

* such outfits are generally keen to gain our business, and 'can be
persuaded' to throw freebies our way. Whilst some see Oracle as 'the
death star', etc, etc, I think their free-stuff might be the most
generous of the bunch. Again, sadly, some learning to be done first...


> Of course, us tools folks are often tasked to write in different
> languages. The "#" symbol is a common, if not universal comment
> designator, and that's why I use it vice language specific things like
> "__author__". Keeping the interpreter and those lines up top really
> makes life easy when I have to sort through a few thousand files to
> track names and the last time it had major modifications.

Once again, an 'advert' for SCM. Such systems offer 'search' facilities,
and certainly track mods, dates, etc! Doing such manually over hundreds
of projects is too much like hard work for this little boy!


As to trying to make one language look like another: nope, not going
there - what is the saying about the British and the Americans being
"divided by the use of a common language"?

My philosophy (such as it is): if you're going to use a tool (language),
then use it - don't play with your food!
(compare Python with JavaScript and 'comments' translate, now the rest
of HTML5 (HTML and CSS)? What about SQL, from earlier discussion 'here'?
It ain't goin' to fly, Wilbur!)

The OP asked about "style". Is there such a thing as a multi-lingual style?


> Some years ago my brother, a better programmer than I, told me about
> "input-output-process" and I try to stick to that. In this case, the
> input is technically just three items, BP, pulse, and timestamp. Usually
> we talk about the two BP numbers as a single unit, and the only reason
> they are separate here is to make parsing easier. The pulse is almost
> ancillary, and the timestamp is for sorting if I ever get to "last X
> number of days", or maybe chart across morning, midday, and evening.
> Because the data is stored in a text file, I can edit (vim, thank you.
> None of that pretty stuff) the timestamp if it needs changing.
> 
> I'm on the fence about f-strings. Ruby has been doing that for years,
> and I usually prefer something that's common-ish between languages.
> Using printf or format() is less visually noisy, for me, than f-strings.
> f-strings have added a lot of usefulness, like Ruby, but much of what I
> do can be done before the print statement.
> 
> Does that make sense?

Perfectly! What is right for you, may not be for me - and vice-versa.

When introducing organisations/teams/colleagues to new tools (or when it
is me being the introduce-ee), my motto is: "nothing succeeds like
success". Thus, if I can show you something working - and working well,
and you say "hey, there's something in that for me", then you'll
adopt/adapt/put-in whatever effort may be necessary. Until then, we're
just having a good time talking...

-- 
Regards,
=dn

From alexkleider at gmail.com  Mon Jun 27 00:56:03 2022
From: alexkleider at gmail.com (Alex Kleider)
Date: Sun, 26 Jun 2022 21:56:03 -0700
Subject: [Tutor] Feedback on coding style
In-Reply-To: <YrjgTv8fuqUfps1T@cskk.homeip.net>
References: <CAMCEyD4aj78B3VSZwUMuXggMYMf0S7eeoJ2+uvrTXiOvMhr3JQ@mail.gmail.com>
 <YrjgTv8fuqUfps1T@cskk.homeip.net>
Message-ID: <CAMCEyD6yEgZE8Xhhat-6hJ9_LcmwVnStoqiij-rg9RbHH1BUsg@mail.gmail.com>

Thanks, Cameron.
Leam already sorted that for me.
a

On Sun, Jun 26, 2022 at 5:45 PM Cameron Simpson <cs at cskk.id.au> wrote:
>
> On 26Jun2022 13:10, Alex Kleider <alexkleider at gmail.com> wrote:
> >Thanks for (so promptly!) getting back to me.
> >I already tried that and was unsuccessful:
> >(perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/
> >(perci) alex at t460:~/Git/LH$ git clone
> >https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py
> >Cloning into 'bp_tracker.py'...
> >fatal: repository
> >'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/'
> >not found
> >(perci) alex at t460:~/Git/LH$
>
> bp_tracker.py is not a repository, so it cannot be cloned.
> You can clone https://github.com/LeamHall/admin_tools instead.
> Or go to
> https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py and
> use the download link on that page.
>
> Cheers,
> Cameron Simpson <cs at cskk.id.au>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor



-- 
alex at kleider.ca  (sent from my current gizmo)

From __peter__ at web.de  Mon Jun 27 04:44:24 2022
From: __peter__ at web.de (Peter Otten)
Date: Mon, 27 Jun 2022 10:44:24 +0200
Subject: [Tutor] problem solving with lists: final (amateur) solution
In-Reply-To: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
Message-ID: <3fd442a8-0578-5683-044c-4aac65ccccde@web.de>

On 25/06/2022 20:45, marcus.luetolf at bluewin.ch wrote:

dn gave you a long list of things that are bad style or outright broken
in your code, a list that may look rather intimidating.


>      def day_1_flights():
>          key_hist = list(history.keys())
>          c_key_hist = key_hist[:]
>          for dummy_i in c_key_hist:
>              print('flights_day_1: ', c_key_hist[:num_in_flight])
>              for key in c_key_hist[:num_in_flight]:
>                  [history[key].append(player)for player in
> c_all_players[0:num_in_flight]]
>              del c_key_hist[:num_in_flight]
>              del c_all_players[0:num_in_flight]


If you want to fix your script and are looking for a starting point I
suggest that you begin with the function above and turn it into a
so-called "pure" one. In simple words this means it returns a result
that depends only on its arguments, does not print anything and does not
modify mutable arguments. Calling the function twice with the same input
will produce the same output twice. Pure functions are the easiest to
reason about and to test.

def day_one_flights(players, num_in_flight):
     """Create flights for the first day.

     >>> day_one_flights(["Peter", "Paul", "Mary", "Marcus"], 2)
     [['Peter', 'Paul'], ['Mary', 'Marcus']]
     """
     ...  # your code

The paragraph in the docstring that looks like an interactive Python
session is a "doctest". You can run it in a terminal window with

py -m doctest players_startlist_commented.py -v

and get a nice report on whether the expected output of the doctests
conforms with the actual, i. e. whether your script is bug free to the
extent that you tested it.



Now allow me to mention one issue specifically:

 >          for dummy_i in c_key_hist:
                ...
 >              del c_key_hist[:num_in_flight]

Here you modify the list you are iterating over. If you are smart enough
to understand what that does you should be too smart to do it ;)
Do not ever write such code!

From marcus.luetolf at bluewin.ch  Mon Jun 27 06:03:36 2022
From: marcus.luetolf at bluewin.ch (marcus.luetolf at bluewin.ch)
Date: Mon, 27 Jun 2022 12:03:36 +0200
Subject: [Tutor] problem solving with lists: final (amateur) solution
In-Reply-To: <3fd442a8-0578-5683-044c-4aac65ccccde@web.de>
References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
 <3fd442a8-0578-5683-044c-4aac65ccccde@web.de>
Message-ID: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch>

Hello Experts, dn and Peter,

I am very grateful for you "scrutinizing" my code and I'll take your
critiques  as incentive for improving it. Nevertheless,  It's quite a
moutfull.
But I'm learning much more about programming and python than by courses over
the internet !!!

Just one replay to Peter's last issue:
I used a copy of the original all_players list because I wanted to keep the
original list unaltered.

It will take awhile til my next "presentation".
Regards and thanks, Marcus.

............................................................................
............................................................................
............................................................................
......

-----Urspr?ngliche Nachricht-----
Von: Tutor <tutor-bounces+marcus.luetolf=bluewin.ch at python.org> Im Auftrag
von Peter Otten
Gesendet: Montag, 27. Juni 2022 10:44
An: tutor at python.org
Betreff: Re: [Tutor] problem solving with lists: final (amateur) solution

On 25/06/2022 20:45, marcus.luetolf at bluewin.ch wrote:

dn gave you a long list of things that are bad style or outright broken in
your code, a list that may look rather intimidating.


>      def day_1_flights():
>          key_hist = list(history.keys())
>          c_key_hist = key_hist[:]
>          for dummy_i in c_key_hist:
>              print('flights_day_1: ', c_key_hist[:num_in_flight])
>              for key in c_key_hist[:num_in_flight]:
>                  [history[key].append(player)for player in 
> c_all_players[0:num_in_flight]]
>              del c_key_hist[:num_in_flight]
>              del c_all_players[0:num_in_flight]


If you want to fix your script and are looking for a starting point I
suggest that you begin with the function above and turn it into a so-called
"pure" one. In simple words this means it returns a result that depends only
on its arguments, does not print anything and does not modify mutable
arguments. Calling the function twice with the same input will produce the
same output twice. Pure functions are the easiest to reason about and to
test.

def day_one_flights(players, num_in_flight):
     """Create flights for the first day.

     >>> day_one_flights(["Peter", "Paul", "Mary", "Marcus"], 2)
     [['Peter', 'Paul'], ['Mary', 'Marcus']]
     """
     ...  # your code

The paragraph in the docstring that looks like an interactive Python session
is a "doctest". You can run it in a terminal window with

py -m doctest players_startlist_commented.py -v

and get a nice report on whether the expected output of the doctests
conforms with the actual, i. e. whether your script is bug free to the
extent that you tested it.



Now allow me to mention one issue specifically:

 >          for dummy_i in c_key_hist:
                ...
 >              del c_key_hist[:num_in_flight]

Here you modify the list you are iterating over. If you are smart enough to
understand what that does you should be too smart to do it ;) Do not ever
write such code!
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From __peter__ at web.de  Tue Jun 28 04:22:10 2022
From: __peter__ at web.de (Peter Otten)
Date: Tue, 28 Jun 2022 10:22:10 +0200
Subject: [Tutor] problem solving with lists: final (amateur) solution
In-Reply-To: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch>
References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
 <3fd442a8-0578-5683-044c-4aac65ccccde@web.de>
 <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch>
Message-ID: <758c874e-53f3-0d4f-12a8-b6da0b074c4e@web.de>

On 27/06/2022 12:03, marcus.luetolf at bluewin.ch wrote:

> Just one replay to Peter's last issue:
> I used a copy of the original all_players list because I wanted to keep the
> original list unaltered.

[untested code below]

Technically you can alter a function

def chunks(items, chunksize):
     result = []
     while items:  # look, ma, no for-loop ;)
         result.append(items[:chunksize])
         del items[:chunksize]
     return result

into one that avoids the side effect of removing the entries from the
items argument by making a copy:

def chunked(items, chunksize):
     return chunks(list(items))

But I doubt that any experienced programmer would write such code. You
are far more likely to see

def chunked(items, chunksize):
     return [
         items[start: start + chunksize]
         for start in range(0, len(items), chunksize)
     ]

While you may find that a little more complex at first a general
strategy that avoids "make a copy, then alter it" in favor of "build a
new object from the current one(s)" makes for cleaner, more readable code.

The two things that made your script hard to understand for me were

(1) name mangling
(2) multiple just-in-case copies

It was not just me, though. The name mangling is what makes it hard to
impossible for you to allow arbitrary player names.

The copies increase the number of variable names and the perceived
complexity of your code. I can tell that you lost track of what was
what, too, as there are places where you have lists that remain
identical over the course of your script, and sometimes mistake a string
for a list and write

players.extend(player)  # /seems/ to work for single-character
                         # player names

Finally: an excellent means to avoid design choices that make your code
hard to maintain is to write tests early on in the design process. If
it's hard to test it's a bad idea.

That's why I suggested doctest in my previous post.

From __peter__ at web.de  Tue Jun 28 04:22:10 2022
From: __peter__ at web.de (Peter Otten)
Date: Tue, 28 Jun 2022 10:22:10 +0200
Subject: [Tutor] problem solving with lists: final (amateur) solution
In-Reply-To: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch>
References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch>
 <3fd442a8-0578-5683-044c-4aac65ccccde@web.de>
 <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch>
Message-ID: <758c874e-53f3-0d4f-12a8-b6da0b074c4e@web.de>

On 27/06/2022 12:03, marcus.luetolf at bluewin.ch wrote:

> Just one replay to Peter's last issue:
> I used a copy of the original all_players list because I wanted to keep the
> original list unaltered.

[untested code below]

Technically you can alter a function

def chunks(items, chunksize):
     result = []
     while items:  # look, ma, no for-loop ;)
         result.append(items[:chunksize])
         del items[:chunksize]
     return result

into one that avoids the side effect of removing the entries from the 
items argument by making a copy:

def chunked(items, chunksize):
     return chunks(list(items))

But I doubt that any experienced programmer would write such code. You 
are far more likely to see

def chunked(items, chunksize):
     return [
         items[start: start + chunksize]
         for start in range(0, len(items), chunksize)
     ]

While you may find that a little more complex at first a general 
strategy that avoids "make a copy, then alter it" in favor of "build a 
new object from the current one(s)" makes for cleaner, more readable code.

The two things that made your script hard to understand for me were

(1) name mangling
(2) multiple just-in-case copies

It was not just me, though. The name mangling is what makes it hard to 
impossible for you to allow arbitrary player names.

The copies increase the number of variable names and the perceived 
complexity of your code. I can tell that you lost track of what was 
what, too, as there are places where you have lists that remain 
identical over the course of your script, and sometimes mistake a string 
for a list and write

players.extend(player)  # /seems/ to work for single-character
                         # player names

Finally: an excellent means to avoid design choices that make your code 
hard to maintain is to write tests early on in the design process. If 
it's hard to test it's a bad idea.

That's why I suggested doctest in my previous post.