From vnthmanoharan at gmail.com  Thu Dec  1 00:28:42 2016
From: vnthmanoharan at gmail.com (vnthmanoharan at gmail.com)
Date: Wed, 30 Nov 2016 21:28:42 -0800 (PST)
Subject: Timer runs only once.
In-Reply-To: <14ce3040-b09f-4b6c-9c52-b72a2a1b20a0@googlegroups.com>
References: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
 <9950fbf4-e096-454c-9d81-7c3057fd20ba@googlegroups.com>
 <14ce3040-b09f-4b6c-9c52-b72a2a1b20a0@googlegroups.com>
Message-ID: <c272c468-4899-4583-ac28-fb28647bfb2a@googlegroups.com>

On Wednesday, 30 November 2016 20:36:15 UTC+5:30, siva gnanam  wrote:
> On Wednesday, November 30, 2016 at 8:11:49 PM UTC+5:30, vnthma... at gmail.com wrote:
> > from threading import Timer
> > 
> > class TestTimer:
> >     def foo(self):
> >         print("hello world")
> >         self.startTimer()
> > 
> >     def startTimer(self):
> >         self.t1 = Timer(5, self.foo)
> >         self.t1.start()
> > 
> > timer = TestTimer()
> > timer.startTimer()
> 
> I think in this example, We are creating Timer object every 5 seconds. So      every time it will span a new Timer. I don't know what happened to the previous timers we created.

It will be garbage collected. The timer object will go out of scope. Python gc is very efficient as it clears the object as and when the obj moves out of scope


From jobmattcon at gmail.com  Thu Dec  1 01:26:30 2016
From: jobmattcon at gmail.com (Ho Yeung Lee)
Date: Wed, 30 Nov 2016 22:26:30 -0800 (PST)
Subject: compile error when using override
Message-ID: <07416672-be69-4329-a3de-d198eb831453@googlegroups.com>

import ast 
from __future__ import division 
from sympy import * 
x, y, z, t = symbols('x y z t') 
k, m, n = symbols('k m n', integer=True) 
f, g, h = symbols('f g h', cls=Function) 
import inspect 

class A:
    @staticmethod
    def __additionFunction__(a1, a2):
        return a1*a2 #Put what you want instead of this
    def __multiplyFunction__(a1, a2):
        return a1*a2+a1 #Put what you want instead of this
    def __init__(self, value):
        self.value = value
    def __add__(self, other):
        return self.__class__.__additionFunction__(self.value, other.value)
    def __mul__(self, other):
        return self.__class__.__multiplyFunction__(self.value, other.value)

solve([A(x)*A(y) + A(-1), A(x) + A(-2)], x, y)


>>> solve([A(x)*A(y) + A(-1), A(x) + A(-2)], x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 12, in __mul__
TypeError: unbound method __multiplyFunction__() must be called with A instance
as first argument (got Symbol instance instead)


import ast 
from __future__ import division 
from sympy import * 
x, y, z, t = symbols('x y z t') 
k, m, n = symbols('k m n', integer=True) 
f, g, h = symbols('f g h', cls=Function) 
import inspect 

 class AA:
    @staticmethod
    def __additionFunction__(a1, a2):
        return a1*a2 #Put what you want instead of this
    def __multiplyFunction__(a1, a2):
        return a1*a2+a1 #Put what you want instead of this
    def __init__(self, value):
        self.value = value
    def __add__(self, other):
        return self.__class__.__additionFunction__(self.value, other.value)
    def __mul__(self, other):
        return self.__class__.__multiplyFunction__(self.value, other.value)


ss = solve(AA)
ss([x*y + -1, x-2], x, y)

>>> ss([x*y + -1, x-2], x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: solve instance has no __call__ method



From varma.nikhil22 at gmail.com  Thu Dec  1 02:14:14 2016
From: varma.nikhil22 at gmail.com (Nikhil Verma)
Date: Thu, 1 Dec 2016 12:44:14 +0530
Subject: Merge Two List of Dict
Message-ID: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>

Hey guys

What is the most optimal and pythonic solution forthis situation

A = [{'person_id': '1', 'adop_count': '2'}, {'person_id': '3',
'adop_count': '4'}]
*len(A) might be above 10L*

B = [{'person_id': '1', 'village_id': '3'}, {'person_id': '3',
'village_id': '4'}]
*len(B) might be above 20L*


OutPut List should be

C = B = [{'adop_count': '2', 'village_id': '3'}, {'adop_count': '4',
'village_id': '4'}]

Thanks in advance


From auriocus at gmx.de  Thu Dec  1 02:53:33 2016
From: auriocus at gmx.de (Christian Gollwitzer)
Date: Thu, 1 Dec 2016 08:53:33 +0100
Subject: Asyncio -- delayed calculation
In-Reply-To: <ea8tcjFodguU1@mid.individual.net>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net>
Message-ID: <o1okrg$pda$1@dont-email.me>

Am 30.11.16 um 22:07 schrieb Gregory Ewing:
> Chris Angelico wrote:
>
>> That's because you're not actually running anything concurrently.
>
> Yes, I know what happens and why. My point is that for
> someone who *doesn't* know, simplistic attempts to
> explain what "await" means can be very misleading.
>
> There doesn't seem to be any accurate way of summarising
> it in a few words. The best we can do seems to be to
> just say "it's a magic word that you have to put in
> front of any call to a function that you defined as
> async".

well that works - but I think it it is possible to explain it, without 
actually understanding what it does behind the scences:

x = foo()
# schedule foo for execution, i.e. put it on a TODO list

await x
# run the TODO list until foo has finished

IMHO coroutines are a simple concept in itself, just that stackful 
programming (call/return) has tainted our minds so much that we have 
trouble figuring out a "function call" which does not "return" in the 
usual sense. The implementation is even more convoluted with the futures 
and promises and whatnot. For simply using that stuff it is not 
important to know how it works.


	Christian


From __peter__ at web.de  Thu Dec  1 03:32:39 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 01 Dec 2016 09:32:39 +0100
Subject: Merge Two List of Dict
References: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
Message-ID: <o1on75$3nc$1@blaine.gmane.org>

Nikhil Verma wrote:

> Hey guys
> 
> What is the most optimal and pythonic solution forthis situation
> 
> A = [{'person_id': '1', 'adop_count': '2'}, {'person_id': '3',
> 'adop_count': '4'}]
> *len(A) might be above 10L*
> 
> B = [{'person_id': '1', 'village_id': '3'}, {'person_id': '3',
> 'village_id': '4'}]
> *len(B) might be above 20L*
> 
> 
> OutPut List should be
> 
> C = B = [{'adop_count': '2', 'village_id': '3'}, {'adop_count': '4',
> 'village_id': '4'}]
> 
> Thanks in advance

Build a lookup table that maps person_id to village_id:

>>> A = [{'person_id': '1', 'adop_count': '2'}, {'person_id': '3',
... 'adop_count': '4'}]
>>> B = [{'person_id': '1', 'village_id': '3'}, {'person_id': '3',
... 'village_id': '4'}]
>>> p2v = {item["person_id"]: item["village_id"] for item in B}
>>> assert len(B) == len(p2v), "duplicate person_id"
>>> import collections
>>> v2a = collections.defaultdict(int)
>>> for item in A:
...     v2a[p2v[item["person_id"]]] += int(item["adop_count"])
... 
>>> [{"adop_count": str(v), "village_id": k} for k, v in v2a.items()]
[{'adop_count': '4', 'village_id': '4'}, {'adop_count': '2', 'village_id': 
'3'}]

If the data stems from a database you can run (untested)

select B.village_id, sum(A.adop_count) from A inner join B on A.person_id = 
B.person_id;



From __peter__ at web.de  Thu Dec  1 04:03:32 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 01 Dec 2016 10:03:32 +0100
Subject: Merge Two List of Dict
References: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
 <o1on75$3nc$1@blaine.gmane.org>
Message-ID: <o1op11$b0c$1@blaine.gmane.org>

Peter Otten wrote:

> If the data stems from a database you can run (untested)
> 
> select B.village_id, sum(A.adop_count) from A inner join B on A.person_id
> = B.person_id;
> 

Oops, I forgot the group-by clause:

select B.village_id, sum(A.adop_count) 
from A inner join B on A.person_id = B.person_id 
group by B.village_id;



From varma.nikhil22 at gmail.com  Thu Dec  1 05:05:09 2016
From: varma.nikhil22 at gmail.com (Nikhil Verma)
Date: Thu, 1 Dec 2016 15:35:09 +0530
Subject: Fwd: Merge Two List of Dict
In-Reply-To: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
References: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
Message-ID: <CABgq=Fx_YicJGKrTCJS=QAcZe3fL3z8za_o7xW54tHJ4sW7A5A@mail.gmail.com>

Just editing the count it was from Indian place value notation.


---------- Forwarded message ----------
From: Nikhil Verma <varma.nikhil22 at gmail.com>
Date: Thu, Dec 1, 2016 at 12:44 PM
Subject: Merge Two List of Dict
To: python-list at python.org


Hey guys

What is the most optimal and pythonic solution forthis situation

A = [{'person_id': '1', 'adop_count': '2'}, {'person_id': '3',
'adop_count': '4'}]
*len(A) might be above 100000*

B = [{'person_id': '1', 'village_id': '3'}, {'person_id': '3',
'village_id': '4'}]
*len(B) might be above 2000000*


OutPut List should be

C = B = [{'adop_count': '2', 'village_id': '3'}, {'adop_count': '4',
'village_id': '4'}]

Thanks in advance





-- 


[image: --]
Nikhil Verma
[image: http://]about.me/nikhil_verma
<http://about.me/nikhil_verma?promo=email_sig>


From steve+python at pearwood.info  Thu Dec  1 06:19:45 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 01 Dec 2016 22:19:45 +1100
Subject: compile error when using override
References: <07416672-be69-4329-a3de-d198eb831453@googlegroups.com>
Message-ID: <58400753$0$1585$c3e8da3$5496439d@news.astraweb.com>

On Thu, 1 Dec 2016 05:26 pm, Ho Yeung Lee wrote:

> import ast
> from __future__ import division

That's not actually your code. That will be a SyntaxError.

Except in the interactive interpreter, "__future__" imports must be the very
first line of code.


> class A:
> ? ? @staticmethod
> ? ? def __additionFunction__(a1, a2):
> ? ? ? ? return a1*a2 #Put what you want instead of this

That cannot work in Python 2, because you are using a "classic"
or "old-style" class. For staticmethod to work correctly, you need to
inherit from object:

class A(object):
    ...


Also, do not use double-underscore names for your own functions or methods.
__NAME__ (two leading and two trailing underscores) are reserved for
Python's internal use. You should not invent your own.

Why do you need this "additionFunction" method for? Why not put this in the
__add__ method?

>   def __add__(self, other):
> ? ? ? return self.__class__.__additionFunction__(self.value, other.value)
> ? def __mul__(self, other):
> ? ? ? return self.__class__.__multiplyFunction__(self.value, other.value)

They should be:

    def __add__(self, other):
        return self.additionFunction(self.value, other.value)

    def __mul__(self, other):
        return self.multiplyFunction(self.value, other.value)

Or better:

    def __add__(self, other):
        return self.value + other.value

    def __mul__(self, other):
        return self.value * other.value



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From python.list at tim.thechases.com  Thu Dec  1 07:08:33 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Thu, 1 Dec 2016 06:08:33 -0600
Subject: Merge Two List of Dict
In-Reply-To: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
References: <CABgq=FwdYGtx4naF_t6i6W-jv7ZcDS4yn+ZeaYKyRdTgWv+mVA@mail.gmail.com>
Message-ID: <20161201060833.388176e2@bigbox.christie.dr>

On 2016-12-01 12:44, Nikhil Verma wrote:
> A = [{'person_id': '1', 'adop_count': '2'}, {'person_id': '3',
> 'adop_count': '4'}]
> *len(A) might be above 10L*
> 
> B = [{'person_id': '1', 'village_id': '3'}, {'person_id': '3',
> 'village_id': '4'}]
> *len(B) might be above 20L*
> 
> 
> OutPut List should be
> 
> C = B = [{'adop_count': '2', 'village_id': '3'}, {'adop_count': '4',
> 'village_id': '4'}]

You omit some details that would help:

- what happened to "person_id" in the results?  Just drop it?

- can duplicates of "person_id" appear in either A or B?  If so, what
  should happen in the output?

- can a "person_id" appear in A or B but not appear in the other?

- is A always just person_id/adop_count and is B always
  person_id/village_id, or can other data appear in (or be absent
  from) each dict?

Your answers would change the implementation details.

-tkc




From p.f.moore at gmail.com  Thu Dec  1 09:03:34 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 1 Dec 2016 06:03:34 -0800 (PST)
Subject: The Case Against Python 3
In-Reply-To: <mailman.46.1480381246.2073.python-list@python.org>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <ea3pq5Fh9nuU1@mid.individual.net>
 <583cc3a1$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmpuJqfvfb4=E3AA3OL4pdbH2HGP7qkTMgibYxWT=iNwzw@mail.gmail.com>
 <mailman.46.1480381246.2073.python-list@python.org>
Message-ID: <e7c59f23-bd4a-49dc-a1c2-f699842a0ebf@googlegroups.com>

On Tuesday, 29 November 2016 01:01:01 UTC, Chris Angelico  wrote:
> So what is it that's trying to read something and is calling an
> f-string a mere string?

gettext.c2py:

    """Gets a C expression as used in PO files for plural forms and returns a
    Python lambda function that implements an equivalent expression.
    """
    # Security check, allow only the "n" identifier
    import token, tokenize
    tokens = tokenize.generate_tokens(io.StringIO(plural).readline)
    try:
        danger = [x for x in tokens if x[0] == token.NAME and x[1] != 'n']
    except tokenize.TokenError:
        raise ValueError('plural forms expression error, maybe unbalanced parenthesis')
    else:
        if danger:
            raise ValueError('plural forms expression could be dangerous')

So the only things that count as DANGER are NAME tokens that aren't "n". That seems pretty permissive...

While I agree that f-strings are more dangerous than people will immediately realise (the mere fact that we call them f-*strings* when they definitely aren't strings is an example of that), the problem here is clearly (IMO) with the sloppy checking in gettext.

Paul


From ned at nedbatchelder.com  Thu Dec  1 11:11:27 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 1 Dec 2016 08:11:27 -0800 (PST)
Subject: The Case Against Python 3
In-Reply-To: <e7c59f23-bd4a-49dc-a1c2-f699842a0ebf@googlegroups.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <ea3pq5Fh9nuU1@mid.individual.net>
 <583cc3a1$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmpuJqfvfb4=E3AA3OL4pdbH2HGP7qkTMgibYxWT=iNwzw@mail.gmail.com>
 <mailman.46.1480381246.2073.python-list@python.org>
 <e7c59f23-bd4a-49dc-a1c2-f699842a0ebf@googlegroups.com>
Message-ID: <14de4986-5d5c-4331-8028-0fa78ed3a324@googlegroups.com>

On Thursday, December 1, 2016 at 9:03:46 AM UTC-5, Paul  Moore wrote:
> While I agree that f-strings are more dangerous than people will immediately realise (the mere fact that we call them f-*strings* when they definitely aren't strings is an example of that), the problem here is clearly (IMO) with the sloppy checking in gettext.


Can you elaborate on the dangers as you see them?

--Ned.


From Cecil at decebal.nl  Thu Dec  1 11:30:31 2016
From: Cecil at decebal.nl (Cecil Westerhof)
Date: Thu, 01 Dec 2016 17:30:31 +0100
Subject: Can json.dumps create multiple lines
Message-ID: <87lgvz4no8.fsf@Equus.decebal.nl>

I started to use json.dumps to put things in a SQLite database. But I
think it would be handy when it would be easy to change the values
manually.

When I have a value dummy which contains:
    ['An array', 'with several strings', 'as a demo']
Then json.dumps(dummy) would generate:
    '["An array", "with several strings", "as a demo"]'
I would prefer when it would generate:
    '[
     "An array",
     "with several strings",
     "as a demo"
     ]'

Is this possible, or do I have to code this myself?

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


From zachary.ware+pylist at gmail.com  Thu Dec  1 11:55:14 2016
From: zachary.ware+pylist at gmail.com (Zachary Ware)
Date: Thu, 1 Dec 2016 10:55:14 -0600
Subject: Can json.dumps create multiple lines
In-Reply-To: <87lgvz4no8.fsf@Equus.decebal.nl>
References: <87lgvz4no8.fsf@Equus.decebal.nl>
Message-ID: <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>

On Thu, Dec 1, 2016 at 10:30 AM, Cecil Westerhof <Cecil at decebal.nl> wrote:
> I would prefer when it would generate:
>     '[
>      "An array",
>      "with several strings",
>      "as a demo"
>      ]'
>
> Is this possible, or do I have to code this myself?

https://docs.python.org/3/library/json.html?highlight=indent#json.dump

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.dumps(["An array", "with several strings", "as a demo"])
'["An array", "with several strings", "as a demo"]'
>>> print(_)
["An array", "with several strings", "as a demo"]
>>> json.dumps(["An array", "with several strings", "as a demo"], indent=0)
'[\n"An array",\n"with several strings",\n"as a demo"\n]'
>>> print(_)
[
"An array",
"with several strings",
"as a demo"
]

I've also seen something about JSON support in SQLite, you may want to
look into that.

-- 
Zach


From gordon at panix.com  Thu Dec  1 11:57:17 2016
From: gordon at panix.com (John Gordon)
Date: Thu, 1 Dec 2016 16:57:17 +0000 (UTC)
Subject: Can json.dumps create multiple lines
References: <87lgvz4no8.fsf@Equus.decebal.nl>
Message-ID: <o1pkpd$t5f$1@reader2.panix.com>

In <87lgvz4no8.fsf at Equus.decebal.nl> Cecil Westerhof <Cecil at decebal.nl> writes:

> I started to use json.dumps to put things in a SQLite database. But I
> think it would be handy when it would be easy to change the values
> manually.

> When I have a value dummy which contains:
>     ['An array', 'with several strings', 'as a demo']
> Then json.dumps(dummy) would generate:
>     '["An array", "with several strings", "as a demo"]'
> I would prefer when it would generate:
>     '[
>      "An array",
>      "with several strings",
>      "as a demo"
>      ]'

json.dumps() has an 'indent' keyword argument, but I believe it only
enables indenting of each whole element, not individual members of a list.

Perhaps something in the pprint module?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From python.list at tim.thechases.com  Thu Dec  1 11:58:24 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Thu, 1 Dec 2016 10:58:24 -0600
Subject: Can json.dumps create multiple lines
In-Reply-To: <87lgvz4no8.fsf@Equus.decebal.nl>
References: <87lgvz4no8.fsf@Equus.decebal.nl>
Message-ID: <20161201105824.4d9cb33d@bigbox.christie.dr>

On 2016-12-01 17:30, Cecil Westerhof wrote:
> When I have a value dummy which contains:
>     ['An array', 'with several strings', 'as a demo']
> Then json.dumps(dummy) would generate:
>     '["An array", "with several strings", "as a demo"]'
> I would prefer when it would generate:
>     '[
>      "An array",
>      "with several strings",
>      "as a demo"
>      ]'
> 
> Is this possible, or do I have to code this myself?

print(json.dumps(['An array', 'with several strings', 'as a demo'],
indent=0))

for the basics of what you ask, though you can change indent= to
indent the contents for readability.

-tkc





From duncan at invalid.invalid  Thu Dec  1 12:10:16 2016
From: duncan at invalid.invalid (duncan smith)
Date: Thu, 1 Dec 2016 17:10:16 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <mailman.29.1480554758.2387.python-list@python.org>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CALvWhxvwxhW5D==UYg5VJo3cdZ_j9=40nabSCm=5KoSYxDajDg@mail.gmail.com>
 <mailman.20.1480528458.2387.python-list@python.org>
 <vxK%z.62849$BE2.28935@fx22.iad>
 <CALvWhxt9zzW2TC9VWG=hg67iCm-cD2zp-J9O+AOBDihiNa+Mig@mail.gmail.com>
 <mailman.29.1480554758.2387.python-list@python.org>
Message-ID: <%PY%z.264392$8F2.40152@fx42.iad>

On 01/12/16 01:12, Chris Kaynor wrote:
> On Wed, Nov 30, 2016 at 4:54 PM, duncan smith <duncan at invalid.invalid> wrote:
>>
>> Thanks. So something like the following might do the job?
>>
>> def _execute(command):
>>     p = subprocess.Popen(command, shell=False,
>>                          stdout=subprocess.PIPE,
>>                          stderr=subprocess.STDOUT,
>>                          close_fds=True)
>>     out_data, err_data = p.communicate()
>>     if err_data:
>>         print err_data
> 
> I did not notice it when I sent my first e-mail (but noted it in my
> second one) that the docstring in to_image is presuming that
> shell=True. That said, as it seems everybody is at a loss to explain
> your issue, perhaps there is some oddity, and if everything appears to
> work with shell=False, it may be worth changing to see if it does fix
> the problem. With other information since provided, it is unlikely,
> however.
> 
> Not specifying the stdin may help, however it will only reduce the
> file handle count by 1 per call (from 2), so there is probably a root
> problem that it will not help.
> 
> I would expect the communicate change to fix the problem, except for
> your follow-up indicating that you had tried that before without
> success.
> 
> Removing the manual stdout.read may fix it, if the problem is due to
> hanging processes, but again, your follow-up indicates thats not the
> problem - you should have zombie processes if that were the case.
> 
> A few new questions that you have not answered (nor have they been
> asked in this thread): How much memory does your system have? Are you
> running a 32-bit or 64-bit Python? Is your Python process being run
> with any additional limitations via system commands (I don't know the
> command, but I know it exists; similarly, if launched from a third
> app, it could be placing limits)?
> 
> Chris
> 

8 Gig, 64 bit, no additional limitations (other than any that might be
imposed by IDLE). In this case the simulation does consume *a lot* of
memory, but that hasn't been the case when I've hit this in the past. I
suppose that could be the issue here. I'm currently seeing if I can
reproduce the problem after adding the p.communicate(), but it seems to
be using more memory than ever (dog slow and up to 5 Gig of swap). In
the meantime I'm going to try to refactor to reduce memory requirements
- and 32 Gig of DDR3 has been ordered. I'll also dig out some code that
generated the same problem before to see if I can reproduce it. Cheers.

Duncan

Duncan


From Joaquin.Alzola at lebara.com  Thu Dec  1 12:17:47 2016
From: Joaquin.Alzola at lebara.com (Joaquin Alzola)
Date: Thu, 1 Dec 2016 17:17:47 +0000
Subject: Can json.dumps create multiple lines
In-Reply-To: <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
References: <87lgvz4no8.fsf@Equus.decebal.nl>
 <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
Message-ID: <DB5PR07MB08065E8A3FB5F0D2B2052D9DF08F0@DB5PR07MB0806.eurprd07.prod.outlook.com>

On Thu, Dec 1, 2016 at 10:30 AM, Cecil Westerhof <Cecil at decebal.nl> wrote:
> I would prefer when it would generate:
>     '[
>      "An array",
>      "with several strings",
>      "as a demo"
>      ]'
>
> Is this possible, or do I have to code this myself?

> https://docs.python.org/3/library/json.html?highlight=indent#json.dump

>>>> json.dumps(["An array", "with several strings", "as a demo"], indent=0)
>'[\n"An array",\n"with several strings",\n"as a demo"\n]'
>>>> print(_)
>[
>"An array",
>"with several strings",
>"as a demo"
>]

As Zac stated the indent:
>>> print(json.dumps(["An array",{"Dummy":{'wop':'dop','dap':'dap'}}, "with several strings", "as a demo"], sort_keys = True, indent=4))
[
    "An array",
    {
        "Dummy": {
            "dap": "dap",
            "wop": "dop"
        }
    },
    "with several strings",
    "as a demo"
]
>>>
This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.


From ian.g.kelly at gmail.com  Thu Dec  1 14:09:45 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Thu, 1 Dec 2016 12:09:45 -0700
Subject: Asyncio -- delayed calculation
In-Reply-To: <o1okrg$pda$1@dont-email.me>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
Message-ID: <CALwzidkb=EQZ2r0Tff-95xj=Mdx1TmCr=DD=H3Z64dS47v98Rw@mail.gmail.com>

On Thu, Dec 1, 2016 at 12:53 AM, Christian Gollwitzer <auriocus at gmx.de> wrote:
> well that works - but I think it it is possible to explain it, without
> actually understanding what it does behind the scences:
>
> x = foo()
> # schedule foo for execution, i.e. put it on a TODO list

This implies that if you never await foo it will still get done at
some point (e.g. when you await something else), which for coroutines
would be incorrect unless you call ensure_future() on it.

Come to think about it, it would probably not be a bad style rule to
consider that when you call something that returns an awaitable, you
should always either await or ensure_future it (or something else that
depends on it). Barring the unusual case where you want to create an
awaitable but *not* immediately schedule it.


From handar94 at gmail.com  Thu Dec  1 16:45:16 2016
From: handar94 at gmail.com (handar94 at gmail.com)
Date: Thu, 1 Dec 2016 13:45:16 -0800 (PST)
Subject: Error In querying Genderize.io. Can someone please help
Message-ID: <af121073-5d8f-4fcd-933e-4f9ba8be279f@googlegroups.com>

import requests
import json
names={'katty','Shean','Rajat'};
for name in names:
    request_string="http://api.genderize.io/?"+name
    r=requests.get(request_string)
    result=json.loads(r.content)


Error-----------
Traceback (most recent call last):
  File "C:/Users/user/PycharmProjects/untitled7/Mis1.py", line 7, in <module>
    result=json.loads(r.content)
  File "C:\Users\user\Anaconda2\lib\json\__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "C:\Users\user\Anaconda2\lib\json\decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\user\Anaconda2\lib\json\decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Can someone please help.


From Cecil at decebal.nl  Thu Dec  1 16:52:24 2016
From: Cecil at decebal.nl (Cecil Westerhof)
Date: Thu, 01 Dec 2016 22:52:24 +0100
Subject: Can json.dumps create multiple lines
References: <87lgvz4no8.fsf@Equus.decebal.nl>
 <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
 <mailman.38.1480611337.2387.python-list@python.org>
Message-ID: <87a8cf48rr.fsf@Equus.decebal.nl>

On Thursday  1 Dec 2016 17:55 CET, Zachary Ware wrote:

> On Thu, Dec 1, 2016 at 10:30 AM, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> I would prefer when it would generate:
>> '[
>> "An array",
>> "with several strings",
>> "as a demo"
>> ]'
>>
>> Is this possible, or do I have to code this myself?
>
> https://docs.python.org/3/library/json.html?highlight=indent#json.dump
>
> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) [GCC 4.2.1
> (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright",
> "credits" or "license" for more information.
>>>> import json
>>>> json.dumps(["An array", "with several strings", "as a demo"])
> '["An array", "with several strings", "as a demo"]'
>>>> print(_)
> ["An array", "with several strings", "as a demo"]
>>>> json.dumps(["An array", "with several strings", "as a demo"],
>>>> indent=0)
> '[\n"An array",\n"with several strings",\n"as a demo"\n]'
>>>> print(_)
> [
> "An array",
> "with several strings",
> "as a demo"
> ]

Works like a charm. Strings can contain newlines also, but then I do
not want a new line, but that works like a charm.

I used:
    cursor.execute('INSERT INTO test (json) VALUES (?)' ,
                   [json.dumps(['An array',
                                'with several strings',
                                'as a demo',
                                'and\none\nwith\na\nnewlines'],
                               indent = 0)])

and that gave exactly what I wanted.

Now I need to convert the database. But that should not be a big
problem.


> I've also seen something about JSON support in SQLite, you may want
> to look into that.

I will do that, but later. I have what I need.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


From Cecil at decebal.nl  Thu Dec  1 17:26:58 2016
From: Cecil at decebal.nl (Cecil Westerhof)
Date: Thu, 01 Dec 2016 23:26:58 +0100
Subject: Can json.dumps create multiple lines
References: <87lgvz4no8.fsf@Equus.decebal.nl>
 <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
 <mailman.38.1480611337.2387.python-list@python.org>
 <87a8cf48rr.fsf@Equus.decebal.nl>
Message-ID: <8737i74765.fsf@Equus.decebal.nl>

On Thursday  1 Dec 2016 22:52 CET, Cecil Westerhof wrote:

> Now I need to convert the database. But that should not be a big
> problem.

I did the conversion with:
    cursor.execute('SELECT tipID FROM tips')
    ids = cursor.fetchall()
    for id in ids:
        id = id[0]
        cursor.execute('SELECT tip from tips WHERE tipID = ?', [id])
        old_value = cursor.fetchone()[0]
        new_value = json.dumps(json.loads(old_value), indent = 0)
        cursor.execute('UPDATE tips SET tip = ? WHERE tipID = ?', [new_value, id])

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


From gordon at panix.com  Thu Dec  1 17:31:46 2016
From: gordon at panix.com (John Gordon)
Date: Thu, 1 Dec 2016 22:31:46 +0000 (UTC)
Subject: Error In querying Genderize.io. Can someone please help
References: <af121073-5d8f-4fcd-933e-4f9ba8be279f@googlegroups.com>
Message-ID: <o1q8ci$a7$1@reader2.panix.com>

In <af121073-5d8f-4fcd-933e-4f9ba8be279f at googlegroups.com> handar94 at gmail.com writes:

> import requests
> import json
> names={'katty','Shean','Rajat'};
> for name in names:
>     request_string="http://api.genderize.io/?"+name
>     r=requests.get(request_string)
>     result=json.loads(r.content)

You're using http: instead of https:, and you're using ?katty instead
of ?name=katty, and therefore the host does not recognize your request
as an API call and redirects you to the normal webpage.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From __peter__ at web.de  Thu Dec  1 17:58:16 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 01 Dec 2016 23:58:16 +0100
Subject: Can json.dumps create multiple lines
References: <87lgvz4no8.fsf@Equus.decebal.nl>
 <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
 <mailman.38.1480611337.2387.python-list@python.org>
 <87a8cf48rr.fsf@Equus.decebal.nl> <8737i74765.fsf@Equus.decebal.nl>
Message-ID: <o1q9u6$umq$1@blaine.gmane.org>

Cecil Westerhof wrote:

> On Thursday  1 Dec 2016 22:52 CET, Cecil Westerhof wrote:
> 
>> Now I need to convert the database. But that should not be a big
>> problem.
> 
> I did the conversion with:
>     cursor.execute('SELECT tipID FROM tips')
>     ids = cursor.fetchall()
>     for id in ids:
>         id = id[0]
>         cursor.execute('SELECT tip from tips WHERE tipID = ?', [id])
>         old_value = cursor.fetchone()[0]
>         new_value = json.dumps(json.loads(old_value), indent = 0)
>         cursor.execute('UPDATE tips SET tip = ? WHERE tipID = ?',
>         [new_value, id])

The sqlite3 module lets you define custom functions written in Python:

db = sqlite3.connect(...)
cs = db.cursor()
 
def convert(s):
    return json.dumps(
        json.loads(s),
        indent=0
    )

db.create_function("convert", 1, convert)
cs.execute("update tips set tip = convert(tip)")




From Cecil at decebal.nl  Thu Dec  1 18:15:15 2016
From: Cecil at decebal.nl (Cecil Westerhof)
Date: Fri, 02 Dec 2016 00:15:15 +0100
Subject: Can json.dumps create multiple lines
References: <87lgvz4no8.fsf@Equus.decebal.nl>
 <CAKJDb-PRFzrKMgf2RACD+4evr99ryMNOacfugnLm9KswqTTZwg@mail.gmail.com>
 <mailman.38.1480611337.2387.python-list@python.org>
 <87a8cf48rr.fsf@Equus.decebal.nl> <8737i74765.fsf@Equus.decebal.nl>
 <o1q9u6$umq$1@blaine.gmane.org>
 <mailman.2.1480633121.2253.python-list@python.org>
Message-ID: <87vav32qd8.fsf@Equus.decebal.nl>

On Thursday  1 Dec 2016 23:58 CET, Peter Otten wrote:

> Cecil Westerhof wrote:
>
>> On Thursday  1 Dec 2016 22:52 CET, Cecil Westerhof wrote:
>>
>>> Now I need to convert the database. But that should not be a big
>>> problem.
>>
>> I did the conversion with:
>> cursor.execute('SELECT tipID FROM tips')
>> ids = cursor.fetchall()
>> for id in ids:
>> id = id[0]
>> cursor.execute('SELECT tip from tips WHERE tipID = ?', [id])
>> old_value = cursor.fetchone()[0]
>> new_value = json.dumps(json.loads(old_value), indent = 0)
>> cursor.execute('UPDATE tips SET tip = ? WHERE tipID = ?',
>> [new_value, id])
>
> The sqlite3 module lets you define custom functions written in
> Python:
>
> db = sqlite3.connect(...)
> cs = db.cursor()
>
> def convert(s):
> return json.dumps(
> json.loads(s),
> indent=0
> )
>
> db.create_function("convert", 1, convert)
> cs.execute("update tips set tip = convert(tip)")

That is a lot better as what I did. Thank you.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


From ned at nedbatchelder.com  Thu Dec  1 18:48:02 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 1 Dec 2016 15:48:02 -0800 (PST)
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <o1ptmr$cbv$3@dont-email.me>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
Message-ID: <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>

On Thursday, December 1, 2016 at 2:31:11 PM UTC-5, DFS wrote:
> After a simple test below, I submit that the above scenario would never 
> occur.  Ever.  The time gap between checking for the file's existence 
> and then trying to open it is far too short for another process to sneak 
> in and delete the file.

It doesn't matter how quickly the first operation is (usually) followed
by the second.  Your process could be swapped out between the two
operations. On a heavily loaded machine, there could be a very long
time between them even if on an average machine, they are executed very
quickly.

For most programs, yes, it probably will never be a problem to check
for existence, and then assume that the file still exists.  But put that
code on a server, and run it a couple of million times, with dozens of
other processes also manipulating files, and you will see failures.

How to best deal with this situation depends on what might happen to the
file, and how you can best coordinate with those other programs. Locks
only help if all the interfering programs also use those same locks. A
popular strategy is to simply use the file, and deal with the error that
happens if the file doesn't exist, though that might not make sense
depending on the logic of the program.  You might have to check that the
file exists, and then also deal with the (slim) possibility that it then
doesn't exist.

--Ned.


From juan0christian at gmail.com  Thu Dec  1 19:07:46 2016
From: juan0christian at gmail.com (Juan C.)
Date: Thu, 1 Dec 2016 22:07:46 -0200
Subject: How to properly retrieve data using requests + bs4 from multiple
 pages in a site?
Message-ID: <CAAp0bGutnU=KGi9gCnZ42cb+FfdE4HVXgfuXddcRBhqsvWz9iA@mail.gmail.com>

I'm a student and my university uses Moodle as their learning management
system (LMS). They don't have Moodle Web Services enabled and won't be
enabling it anytime soon, at least for students. The university programs
have the following structure, for example:

1. Bachelor's Degree in Computer Science (duration: 8 semesters)

1.1. Unit 01: Mathematics Fundamental (duration: 1 semester)
1.1.1. Algebra I (first 3 months)
1.1.2. Algebra II (first 3 months)
1.1.3. Calculus I (last 3 months)
1.1.4. Calculus II (last 3 months)
1.1.5. Unit Project (throughout the semester)

1.2. Unit 02: Programming (duration: 1 semester)
1.2.1. Programming Logic (first 3 months)
1.2.2. Data Modelling with UML (first 3 months)
1.2.3. Python I (last 3 months)
1.2.4. Python II (last 3 months)
1.2.5. Unit Project (throughout the semester)

Each course/project have a bunch of assignments + one final assignment.
This goes on, totalizing 8 (eight) units, which will make up for a 4-year
program. I'm building my own client-side Moodle API to be consumed by my
scripts. Currently I'm using 'requests' + 'bs4' to do the job. My code:

package moodle/

user.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from .program import Program
import requests


class User:
   _AUTH_URL = 'http://lms.university.edu/moodle/login/index.php'

   def __init__(self, username, password, program_id):
      self.username = username
      self.password = password
      session = requests.session()
      session.post(self._AUTH_URL, {"username": username, "password":
password})
      self.program = Program(program_id=program_id, session=session)

   def __str__(self):
      return self.username + ':' + self.password

   def __repr__(self):
      return '<User %s>' % self.username

   def __eq__(self, other):
      if isinstance(other, self):
         return self.username == other.username
      else:
         return False

==========

program.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from .unit import Unit
from bs4 import BeautifulSoup


class Program:
   _PATH = 'http://lms.university.edu/moodle/course/index.php?categoryid='

   def __init__(self, program_id, session):
      response = session.get(self._PATH + str(program_id))
      soup = BeautifulSoup(response.text, 'html.parser')

      self.name = soup.find('ul',
class_='breadcrumb').find_all('li')[-2].text.replace('/', '').strip()
      self.id = program_id
      self.units = [Unit(int(item['data-categoryid']), session) for item in
soup.find_all('div', {'class': 'category'})]

   def __str__(self):
      return self.name

   def __repr__(self):
      return '<Program %s (%s)>' % (self.name, self.id)

   def __eq__(self, other):
      if isinstance(other, self):
         return self.id == other.id
      else:
         return False

==========

unit.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from .course import Course
from bs4 import BeautifulSoup


class Unit:
   _PATH = 'http://lms.university.edu/moodle/course/index.php?categoryid='

   def __init__(self, unit_id, session):
      response = session.get(self._PATH + str(unit_id))
      soup = BeautifulSoup(response.text, 'html.parser')

      self.name = soup.find('ul',
class_='breadcrumb').find_all('li')[-1].text.replace('/', '').strip()
      self.id = unit_id
      self.courses = [Course(int(item['data-courseid']), session) for item
in soup.find_all('div', {'class': 'coursebox'})]

   def __str__(self):
      return self.name

   def __repr__(self):
      return '<Unit %s (%s)>' % (self.name, self.id)

   def __eq__(self, other):
      if isinstance(other, self):
         return self.id == other.id
      else:
         return False

==========

course.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-


from .assignment import Assignment
import re
from bs4 import BeautifulSoup


class Course:
   _PATH = 'http://lms.university.edu/moodle/course/view.php?id='

   def __init__(self, course_id, session):
      response = session.get(self._PATH + str(course_id))
      soup = BeautifulSoup(response.text, 'html.parser')

      self.name = soup.find('h1').text
      self.id = course_id
      self.assignments = [Assignment(int(item['href'].split('id=')[-1]),
session) for item in
         soup.find_all('a', href=re.compile(r'http://lms
\.university\.edu/moodle/mod/assign/view.php\?id=.*'))]

   def __str__(self):
      return self.name

   def __repr__(self):
      return '<Course %s (%s)>' % (self.name, self.id)

   def __eq__(self, other):
      if isinstance(other, self):
         return self.id == other.id
      else:
         return False

==========

assignment.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from bs4 import BeautifulSoup


class Assignment:
   _PATH = 'http://lms.university.edu/moodle/mod/assign/view.php?id='

   def __init__(self, assignment_id, session):
      response = session.get(self._PATH + str(assignment_id))
      soup = BeautifulSoup(response.text, 'html.parser')

      self.name = soup.find('h2').text
      self.id = assignment_id
      self.sent = soup.find('td', {'class': 'submissionstatussubmitted'})
is not None
      self.graded = soup.find('td', {'class': 'submissiongraded'}) is not
None
      # more attributes will go here, like rubrics, due_date, etc. That's a
work in progress.

   def __str__(self):
      return self.name

   def __repr__(self):
      return '<Assignment %s (%s)>' % (self.name, self.id)

   def __eq__(self, other):
      if isinstance(other, self):
         return self.id == other.id
      else:
         return False

==========

test.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from moodle.user import User

my_user = User('john.smith', '3uper$secret', 12)

print(my_user)  # print: john.smith:3uper$secret

print(my_user.program)  # print: Computer Science

print(my_user.program.units)  # print: [<Unit Math Fundamentals (59)>,
<Unit Programming (102)>]

print(my_user.program.units[-1].courses)  # print: [<Course Programming
Logic (666)>, <Course Data Modeling (667)>, <Course Python 1 (668)>,
<Course Python 2 (669)>, <Course Project (670)>]

print(my_user.program.units[-1].courses[-1].assignments)  # print:
[<Assignment A1 [mandatory] (40817)>, <Assignment A2 (40824)>, <Assignment
A3 [mandatory] (40831)>, <Assignment A4 (40838)>, <Assignment A5
[mandatory] (40845)>, <Assignment Final Assignment [mandatory] (40882)>]

==========

It works, but it has a big issue: it gets all data from all
units/courses/assignments at the same time, and this isn't very useful as I
don't care about data from units from 1-2 years ago. How can I change the
logic so it just gets the data I need at a given moment? For example, I may
need to dump data for an entire unit, or just one course, or maybe even
just one assignment. How can I achieve this behavior? Another "issue", I
feel like handing my 'session' that I instantiated at user.py to program,
then unit, then course and then assignment is a poor design, how can I make
it better?

Any other suggestions are welcome.


From jobmattcon at gmail.com  Thu Dec  1 21:35:44 2016
From: jobmattcon at gmail.com (Ho Yeung Lee)
Date: Thu, 1 Dec 2016 18:35:44 -0800 (PST)
Subject: compile error when using override
In-Reply-To: <58400753$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <07416672-be69-4329-a3de-d198eb831453@googlegroups.com>
 <58400753$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <48fc3a2a-4cae-4e03-bc1a-c23151ec3ff2@googlegroups.com>

from __future__ import division 
import ast 
from sympy import * 
x, y, z, t = symbols('x y z t') 
k, m, n = symbols('k m n', integer=True) 
f, g, h = symbols('f g h', cls=Function) 
import inspect 
def op2(a,b): 
    return a*b+a 

class AA(object):
    @staticmethod
    def __additionFunction__(a1, a2):
        return a1*a2 #Put what you want instead of this
    def __multiplyFunction__(a1, a2):
        return a1*a2+a1 #Put what you want instead of this
    def __divideFunction__(a1, a2):
        return a1*a1*a2 #Put what you want instead of this
    def __init__(self, value):
        self.value = value
    def __add__(self, other):
        return self.value*other.value
    def __mul__(self, other):
        return self.value*other.value + other.value
    def __div__(self, other):
        return self.value*other.value*other.value

solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y)

>>> class AA(object):
...     @staticmethod
...     def __additionFunction__(a1, a2):
...         return a1*a2 #Put what you want instead of this
...     def __multiplyFunction__(a1, a2):
...         return a1*a2+a1 #Put what you want instead of this
...     def __divideFunction__(a1, a2):
...         return a1*a1*a2 #Put what you want instead of this
...     def __init__(self, value):
...         self.value = value
...     def __add__(self, other):
...         return self.value*other.value
...     def __mul__(self, other):
...         return self.value*other.value + other.value
...     def __div__(self, other):
...         return self.value*other.value*other.value
...
>>> solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'Add' and 'AA'


On Thursday, December 1, 2016 at 7:19:58 PM UTC+8, Steve D'Aprano wrote:
> On Thu, 1 Dec 2016 05:26 pm, Ho Yeung Lee wrote:
> 
> > import ast
> > from __future__ import division
> 
> That's not actually your code. That will be a SyntaxError.
> 
> Except in the interactive interpreter, "__future__" imports must be the very
> first line of code.
> 
> 
> > class A:
> > ? ? @staticmethod
> > ? ? def __additionFunction__(a1, a2):
> > ? ? ? ? return a1*a2 #Put what you want instead of this
> 
> That cannot work in Python 2, because you are using a "classic"
> or "old-style" class. For staticmethod to work correctly, you need to
> inherit from object:
> 
> class A(object):
>     ...
> 
> 
> Also, do not use double-underscore names for your own functions or methods.
> __NAME__ (two leading and two trailing underscores) are reserved for
> Python's internal use. You should not invent your own.
> 
> Why do you need this "additionFunction" method for? Why not put this in the
> __add__ method?
> 
> >   def __add__(self, other):
> > ? ? ? return self.__class__.__additionFunction__(self.value, other.value)
> > ? def __mul__(self, other):
> > ? ? ? return self.__class__.__multiplyFunction__(self.value, other.value)
> 
> They should be:
> 
>     def __add__(self, other):
>         return self.additionFunction(self.value, other.value)
> 
>     def __mul__(self, other):
>         return self.multiplyFunction(self.value, other.value)
> 
> Or better:
> 
>     def __add__(self, other):
>         return self.value + other.value
> 
>     def __mul__(self, other):
>         return self.value * other.value
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.



From best_lay at yahoo.com  Thu Dec  1 22:09:20 2016
From: best_lay at yahoo.com (Wildman)
Date: Thu, 01 Dec 2016 21:09:20 -0600
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <ahit3ctqfvlrghkfsdau4khbv41brmkk0t@4ax.com>
 <mailman.11.1480510497.2387.python-list@python.org>
Message-ID: <EPmdnWsCX899eN3FnZ2dnUU7-T-dnZ2d@giganews.com>

On Wed, 30 Nov 2016 07:54:45 -0500, Dennis Lee Bieber wrote:

> On Tue, 29 Nov 2016 22:01:51 -0600, Wildman via Python-list
> <python-list at python.org> declaimed the following:
> 
>>I really appreciate your reply.  Your suggestion fixed that
>>problem, however, a new error appeared.  I am doing some
>>research to try to figure it out but no luck so far.
>>
>>Traceback (most recent call last):
>>  File "./ifaces.py", line 33, in <module>
>>    ifs = all_interfaces()
>>  File "./ifaces.py", line 21, in all_interfaces
>>    name = namestr[i:i+16].split('\0', 1)[0]
>>TypeError: Type str doesn't support the buffer API
> 
> 	The odds are good that this is the same class of problem -- you are
> providing a Unicode string to a procedure that wants a byte-string (or vice
> versa)
> 
> https://docs.python.org/3/library/array.html?highlight=tostring#array.array.tostring

That helped.  Thanks.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Thu Dec  1 22:16:17 2016
From: best_lay at yahoo.com (Wildman)
Date: Thu, 01 Dec 2016 21:16:17 -0600
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <vg3wpfl9m6x.fsf@coffee.modeemi.fi>
 <mailman.9.1480509554.2387.python-list@python.org>
Message-ID: <EPmdnWoCX88cet3FnZ2dnUU7-T8AAAAA@giganews.com>

On Wed, 30 Nov 2016 14:39:02 +0200, Anssi Saari wrote:

> There'll be a couple more issues with the printing but they should be
> easy enough.

I finally figured it out, I think.  I'm not sure if my changes are
what you had in mind but it is working.  Below is the updated code.
Thank you for not giving me the answer.  I was a good learning
experience for me and that was my purpose in the first place.

def format_ip(addr):
    return str(int(addr[0])) + '.' + \ # replace ord() with int()
           str(int(addr[1])) + '.' + \
           str(int(addr[2])) + '.' + \
           str(int(addr[3]))

ifs = all_interfaces()
for i in ifs:         # added decode("utf-8")
    print("%12s   %s" % (i[0].decode("utf-8"), format_ip(i[1])))

Thanks again!

-- 
<Wildman> GNU/Linux user #557453
May the Source be with you.


From ned at nedbatchelder.com  Thu Dec  1 22:39:57 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 1 Dec 2016 19:39:57 -0800 (PST)
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <o1qf0h$bhu$1@dont-email.me>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
Message-ID: <b54fe8a8-e6e2-4d18-8031-534aefa0fd84@googlegroups.com>

On Thursday, December 1, 2016 at 7:26:18 PM UTC-5, DFS wrote:
> On 12/01/2016 06:48 PM, Ned Batchelder wrote:
> > On Thursday, December 1, 2016 at 2:31:11 PM UTC-5, DFS wrote:
> >> After a simple test below, I submit that the above scenario would never
> >> occur.  Ever.  The time gap between checking for the file's existence
> >> and then trying to open it is far too short for another process to sneak
> >> in and delete the file.
> >
> > It doesn't matter how quickly the first operation is (usually) followed
> > by the second.  Your process could be swapped out between the two
> > operations. On a heavily loaded machine, there could be a very long
> > time between them
> 
> 
> How is it possible that the 'if' portion runs, then 44/100,000ths of a 
> second later my process yields to another process which deletes the 
> file, then my process continues.

A modern computer is running dozens or hundreds (or thousands!) of
processes "all at once". How they are actually interleaved on the
small number of actual processors is completely unpredictable. There
can be an arbitrary amount of time passing between any two processor
instructions.

I'm assuming you've measured this program on your own computer, which
was relatively idle at the moment.  This is hardly a good stress test
of how the program might execute under more burdened conditions.

> 
> Is that governed by the dreaded GIL?
> 
> "The mechanism used by the CPython interpreter to assure that only one 
> thread executes Python bytecode at a time."
> 
> But I see you posted a stack-overflow answer:
> 
> "In the case of CPython's GIL, the granularity is a bytecode 
> instruction, so execution can switch between threads at any bytecode."
> 
> Does that mean "chars=f.read().lower()" could get interrupted between 
> the read() and the lower()?

Yes.  But even more importantly, the Python interpreter is itself a
C program, and it can be interrupted between any two instructions, and
another program on the computer could run instead.  That other program
can fiddle with files on the disk.

> 
> I read something interesting last night:
> https://www.jeffknupp.com/blog/2012/03/31/pythons-hardest-problem/
> 
> "In the new GIL, a hard timeout is used to instruct the current thread 
> to give up the lock. When a second thread requests the lock, the thread 
> currently holding it is compelled to release it after 5ms (that is, it 
> checks if it needs to release it every 5ms)."
> 
> With a 5ms window, it seems the following code would always protect the 
> file from being deleted between lines 4 and 5.
> 
> --------------------------------
> 1 import os,threading
> 2 f_lock=threading.Lock()
> 3 with f_lock:
> 4   if os.path.isfile(filename):
> 5     with open(filename,'w') as f:
> 6       process(f)
> --------------------------------
> 

You seem to be assuming that the program that might delete the file
is the same program trying to read the file.  I'm not assuming that.
My Python program might be trying to read the file at the same time
that a cron job is running a shell script that is trying to delete
the file.

> Also, this is just theoretical (I hope).  It would be terrible system 
> design if all those dozens of processes were reading and writing and 
> deleting the same file.

If you can design your system so that you know for sure no one else
is interested in fiddling with your file, then you have an easier
problem.  So far, that has not been shown to be the case. I'm
talking more generally about a program that can't assume those
constraints.

--Ned.


From steve+python at pearwood.info  Thu Dec  1 23:47:45 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 02 Dec 2016 15:47:45 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
Message-ID: <5840fcf3$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Fri, 2 Dec 2016 11:26 am, DFS wrote:

> On 12/01/2016 06:48 PM, Ned Batchelder wrote:
>> On Thursday, December 1, 2016 at 2:31:11 PM UTC-5, DFS wrote:
>>> After a simple test below, I submit that the above scenario would never
>>> occur.  Ever.  The time gap between checking for the file's existence
>>> and then trying to open it is far too short for another process to sneak
>>> in and delete the file.
>>
>> It doesn't matter how quickly the first operation is (usually) followed
>> by the second.  Your process could be swapped out between the two
>> operations. On a heavily loaded machine, there could be a very long
>> time between them
> 
> 
> How is it possible that the 'if' portion runs, then 44/100,000ths of a
> second later my process yields to another process which deletes the
> file, then my process continues.
> 
> Is that governed by the dreaded GIL?

No, that has nothing to do with the GIL. It is because the operating 
system is a preemptive multi-processing operating system. All modern OSes 
are: Linux, OS X, Windows.

Each program that runs, including the OS itself, is one or more processes.
Typically, even on a single-user desktop machine, you will have dozens of
processes running simultaneously.

Every so-many clock ticks, the OS pauses whatever process is running, 
more-or-less interrupting whatever it was doing, passes control on to 
another process, then the next, then the next, and so on. The application 
doesn't have any control over this, it can be paused at any time, 
normally just for a small fraction of a second, but potentially for 
seconds or minutes at a time if the system is heavily loaded.



> "The mechanism used by the CPython interpreter to assure that only one
> thread executes Python bytecode at a time."
> 
> But I see you posted a stack-overflow answer:
> 
> "In the case of CPython's GIL, the granularity is a bytecode
> instruction, so execution can switch between threads at any bytecode."
> 
> Does that mean "chars=f.read().lower()" could get interrupted between
> the read() and the lower()?

Yes, but don't think about Python threads. Think about the OS.

I'm not an expert on the low-level hardware details, so I welcome
correction, but I think that you can probably expect that the OS can
interrupt code execution between any two CPU instructions. Something like
str.lower() is likely to be thousands of CPU instructions, even for a small
string.


[...]
> With a 5ms window, it seems the following code would always protect the
> file from being deleted between lines 4 and 5.
> 
> --------------------------------
> 1 import os,threading
> 2 f_lock=threading.Lock()
> 3 with f_lock:
> 4   if os.path.isfile(filename):
> 5     with open(filename,'w') as f:
> 6       process(f)
> --------------------------------
> 
> 
> 
>> even if on an average machine, they are executed very quickly.

Absolutely not. At least on Linux, locks are advisory, not mandatory. Here
are a pair of scripts that demonstrate that. First, the well-behaved script
that takes out a lock:

# --- locker.py ---
import os, threading, time

filename = 'thefile.txt'
f_lock = threading.Lock()

with f_lock:
    print '\ntaking lock'
    if os.path.isfile(filename):
        print filename, 'exists and is a file'
        time.sleep(10)
        print 'lock still active'
        with open(filename,'w') as f:
            print f.read()

# --- end ---


Now, a second script which naively, or maliciously, just deletes the file:

# --- bandit.py ---
import os, time
filename = 'thefile.txt'
time.sleep(1)
print 'deleting file, mwahahahaha!!!'
os.remove(filename)
print 'deleted'

# --- end ---



Now, I run them both simultaneously:

[steve at ando thread-lock]$ touch thefile.txt # ensure file exists
[steve at ando thread-lock]$ (python locker.py &) ; (python bandit.py &)
[steve at ando thread-lock]$ 
taking lock
thefile.txt exists and is a file
deleting file, mwahahahaha!!!
deleted
lock still active
Traceback (most recent call last):
  File "locker.py", line 14, in <module>
    print f.read()
IOError: File not open for reading



This is on Linux. Its possible that Windows behaves differently, and I don't
know how to run a command in the background in command.com or cmd.exe or
whatever you use on Windows.


[...]
> Also, this is just theoretical (I hope).  It would be terrible system
> design if all those dozens of processes were reading and writing and
> deleting the same file.

It is not theoretical. And it's not a terrible system design, in the sense
that the alternatives are *worse*.

* Turn the clock back to the 1970s and 80s with single-processing 
  operating systems? Unacceptable -- even primitive OSes like DOS 
  and Mac System 5 needed to include some basic multiprocessing 
  capability.

- And what are servers supposed to do in this single-process world?

- Enforce mandatory locks? A great way for malware or hostile users
  to perform Denial Of Service attacks.

Even locks being left around accidentally can be a real pain: Windows users
can probably tell you about times that a file has been accidentally left
open by buggy applications, and there's nothing you can do to unlock it
short of rebooting. Unacceptable for a server, and pain in the rear even for
a desktop.

- Make every file access go through a single scheduling application
  which ensures there are no clashes? Probably very hard to write,
  and would probably kill performance. Imagine you cannot even check
  the existence of a 4GB file until its finished copying onto a USB
  stick... 



The cost of allowing two programs to run at the same time is that 
sometimes they will both want to do something to the same file.

Fundamentally though, the solution here is quite simple: don't rely on 
"Look Before You Leap" checks any time you have shared data, and the 
file system is shared data. If you want *reliable* code, you MUST use a 
try...except block to recover from file system errors.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, 
and sure enough, things got worse.



From steve+python at pearwood.info  Fri Dec  2 00:02:04 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 02 Dec 2016 16:02:04 +1100
Subject: compile error when using override
References: <07416672-be69-4329-a3de-d198eb831453@googlegroups.com>
 <58400753$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <48fc3a2a-4cae-4e03-bc1a-c23151ec3ff2@googlegroups.com>
Message-ID: <5841004e$0$1619$c3e8da3$5496439d@news.astraweb.com>

On Fri, 2 Dec 2016 01:35 pm, Ho Yeung Lee wrote:

> from __future__ import division
> import ast
> from sympy import *
> x, y, z, t = symbols('x y z t')
> k, m, n = symbols('k m n', integer=True)
> f, g, h = symbols('f g h', cls=Function)
> import inspect

Neither ast nor inspect is used. Why import them?

The only symbols you are using are x and y.


> def op2(a,b):
>     return a*b+a

This doesn't seem to be used. Get rid of it.


> class AA(object):
>     @staticmethod
>     def __additionFunction__(a1, a2):
>         return a1*a2 #Put what you want instead of this
>     def __multiplyFunction__(a1, a2):
>         return a1*a2+a1 #Put what you want instead of this
>     def __divideFunction__(a1, a2):
>         return a1*a1*a2 #Put what you want instead of this

None of those methods are used. Get rid of them.

>     def __init__(self, value):
>         self.value = value
>     def __add__(self, other):
>         return self.value*other.value

Sorry, you want AA(5) + AA(2) to return 10?

>     def __mul__(self, other):
>         return self.value*other.value + other.value
>     def __div__(self, other):
>         return self.value*other.value*other.value
> 
> solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y)

I don't understand what you are trying to do here. What result are you
execting?

Maybe you just want this?

from sympy import solve, symbols
x, y = symbols('x y')
print( solve([x*y - 1, x - 2], x, y) )

which prints the result:
[(2, 1/2)]


Perhaps if you explain what you are trying to do, we can help better.

But please, cut down your code to only code that is being used!




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Fri Dec  2 00:41:17 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 02 Dec 2016 16:41:17 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
Message-ID: <5841097f$0$1612$c3e8da3$5496439d@news.astraweb.com>

On Fri, 2 Dec 2016 11:26 am, DFS wrote:

>> For most programs, yes, it probably will never be a problem to check
>> for existence, and then assume that the file still exists. ?But put that
>> code on a server, and run it a couple of million times, with dozens of
>> other processes also manipulating files, and you will see failures.
> 
> 
> If it's easy for you, can you write some short python code to simulate
> that?

Run these scripts simultaneously inside the same directory, and you will see
a continual stream of error messages:

# -- a.py -- 
filename = 'data'
import os, time

def run():
    if os.path.exists(filename):
        with open(filename):
            pass
    else:
        print('file is missing!')
        # re-create it
        with open(filename, 'w'):
            pass

while True:
    try:
        run()
    except IOError:
        pass
    time.sleep(0.05)



# -- b.py --
filename = 'data'
import os, time

while True:
    try:
        os.remove(filename)
    except OSError:
        pass
    time.sleep(0.05)




The time.sleep() calls are just to slow them down slightly. You can leave
them out if you like.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From magdalena.guz80 at gmail.com  Fri Dec  2 02:01:47 2016
From: magdalena.guz80 at gmail.com (Gus_G)
Date: Thu, 1 Dec 2016 23:01:47 -0800 (PST)
Subject: What do you think: good idea to launch a marketplace on python+django?
Message-ID: <9ee77ca5-f2c8-48ef-bae8-bc4a1feef483@googlegroups.com>

Hello, what do you think about building a marketplace website on connection of python+django? End effect-side should look and work similar to these: https://zoptamo.com/uk/s-abs-c-uk, https://www.ownerdirect.com/ . What are your opinions on this idea? Maybe there is other, better way to build it?


From steve+python at pearwood.info  Fri Dec  2 08:59:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 03 Dec 2016 00:59:07 +1100
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
Message-ID: <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>

On Thu, 1 Dec 2016 06:53 pm, Christian Gollwitzer wrote:

> well that works - but I think it it is possible to explain it, without
> actually understanding what it does behind the scences:
> 
> x = foo()
> # schedule foo for execution, i.e. put it on a TODO list
> 
> await x
> # run the TODO list until foo has finished

Nope, sorry, that doesn't help even a little bit. For starters, it doesn't
work: it is a syntax error.

py> async def foo():
...     return 1
...
py> x = foo()
py> await x
  File "<stdin>", line 1
    await x
          ^
SyntaxError: invalid syntax


But even if it did work, why am I waiting for the TODO list to finish?
Doesn't that mean I'm now blocking, which goes completely against the idea
of writing non-blocking asynchronous code? If I wanted to block waiting for
x, I'd just make it a regular, easy-to-understand, synchronous function.

Besides, where does x stash it's result? In a global variable? What if
another async routine messes with the same global?

My first impressions on this is that we have a couple of good models for
preemptive parallelism, threads and processes, both of which can do
everything that concurrency can do, and more, and both of which are
significantly easier to understand too.

So why do we need asyncio? What is it actually good for?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From marko at pacujo.net  Fri Dec  2 09:19:29 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 02 Dec 2016 16:19:29 +0200
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
 <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87fum6v2fi.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> py> await x
>   File "<stdin>", line 1
>     await x
>           ^
> SyntaxError: invalid syntax

"await" is only allowed inside a coroutine.

> So why do we need asyncio? What is it actually good for?

Asyncio is a form of cooperative multitasking. It presents a framework
of "fake threads". The objective and programming model is identical to
that of threads.

As to why Python should need a coroutine framework, you can answer it
from two angles:

  1. Why does one need cooperative multitasking?

  2. Why should one use coroutines to implement cooperative
     multitasking?


1. Cooperative multitasking has made its way in Java ("NIO") and C#
(async/await). It has come about as enterprise computing realized the
multithreading model of the 1990's was shortsighted. In particular, it
wasn't scalable. Enterprise solutions collapsed under the weight of tens
of thousands of threads. Stack space ran out and schedulers became slow.
<URL: https://en.wikipedia.org/wiki/C10k_problem>

2. I have always been into asynchronous programming (cooperative
multitasking), but coroutines are far from my favorite programming
model. I am guessing Guido introduced them to Python because:

 * C# has them (me too!).

 * They have a glorious computer scientific past (CSP, emperor's new I/O
   framework).

 * They look like threads.

 * They were already there in the form of generators (low-hanging
   fruit).

And, maybe most importantly:

 * Twisted (et al) had needed an event-driven framework but Python
   didn't have one out of the box (<URL:
   https://lwn.net/Articles/692254/>).


Marko


From frank at chagford.com  Fri Dec  2 09:26:30 2016
From: frank at chagford.com (Frank Millman)
Date: Fri, 2 Dec 2016 16:26:30 +0200
Subject: Asyncio -- delayed calculation
In-Reply-To: <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
 <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o1s0am$lso$1@blaine.gmane.org>

"Steve D'Aprano"  wrote in message 
news:58417e2d$0$1612$c3e8da3$5496439d at news.astraweb.com...
>
> My first impressions on this is that we have a couple of good models for
> preemptive parallelism, threads and processes, both of which can do
> everything that concurrency can do, and more, and both of which are
> significantly easier to understand too.
>
> So why do we need asyncio? What is it actually good for?
>

As I have mentioned, my use-case is a multi-user client/server system.

The traditional approach used to be to use threads to allow multiple users 
to work concurrently.

Then Twisted made a strong case for an asynchronous approach. One of their 
claims (which I have no reason to doubt) was that, because each user 
'session' spends most of its time waiting for something - keyboard input, 
reply from database, etc - their approach allows hundreds of concurrent 
users, something that I believe would not be possible with threading or 
multi-processing.

Frank Millman




From marko at pacujo.net  Fri Dec  2 09:42:13 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 02 Dec 2016 16:42:13 +0200
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
 <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>
 <o1s0am$lso$1@blaine.gmane.org>
 <mailman.4.1480688817.2253.python-list@python.org>
Message-ID: <87bmwuv1dm.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:
> Then Twisted made a strong case for an asynchronous approach. One of
> their claims (which I have no reason to doubt) was that, because each
> user 'session' spends most of its time waiting for something -
> keyboard input, reply from database, etc - their approach allows
> hundreds of concurrent users, something that I believe would not be
> possible with threading or multi-processing.

You don't need asyncio to do event-driven programming in Python. I have
programmed several event-driven applications in Python (and even more of
them in other languages) without using asyncio (you only need
select.epoll()).

My favorite model is the so-called "callback hell" with explicit finite
state machines. That tried-and-true model has always been used with
parallel, concurrent and network programming as well as user-interface
programming. Your code should closely resemble this:

   <URL: http://www.bayfronttechnologies.com/capesdl.gif>


Multiprocessing is also an important tool for compartmentalizing and
parallelizing functionality. Threads are mainly suitable for turning
obnoxious blocking APIs into nonblocking ones, but even there, I would
prefer to give that job to separate processes.


Marko


From rosuav at gmail.com  Fri Dec  2 09:54:14 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 3 Dec 2016 01:54:14 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <o1s0am$lso$1@blaine.gmane.org>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net> <o1okrg$pda$1@dont-email.me>
 <58417e2d$0$1612$c3e8da3$5496439d@news.astraweb.com>
 <o1s0am$lso$1@blaine.gmane.org>
Message-ID: <CAPTjJmrkzUMXTw_548_t3MUdrm_HPp7HEy4eQT_3kfTsaEeovw@mail.gmail.com>

On Sat, Dec 3, 2016 at 1:26 AM, Frank Millman <frank at chagford.com> wrote:
> Then Twisted made a strong case for an asynchronous approach. One of their
> claims (which I have no reason to doubt) was that, because each user
> 'session' spends most of its time waiting for something - keyboard input,
> reply from database, etc - their approach allows hundreds of concurrent
> users, something that I believe would not be possible with threading or
> multi-processing.

I'm not sure that "hundreds" would be a problem - I've had processes
with hundreds of threads before - but if you get up to hundreds of
*thousands* of threads, then yes, threads start to be a problem. When
you have that sort of traffic (that's multiple requests per
millisecond, or over a million requests per minute), you have to think
about throughput and efficiency, not just simplicity. But for low
traffic environments (single digits of requests per second), threads
work just fine. Actually, for traffic levels THAT low, you could
probably serialize your requests and nobody would notice. That's the
simplest of all, but it scales pretty terribly :)

Not everyone needs the full power of asyncio, so not everyone will
really see the point. I like that it's available to everyone, though.
If you need it, it's there.

ChrisA


From grant.b.edwards at gmail.com  Fri Dec  2 10:11:18 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 2 Dec 2016 15:11:18 +0000 (UTC)
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <vg3wpfl9m6x.fsf@coffee.modeemi.fi>
 <mailman.9.1480509554.2387.python-list@python.org>
 <EPmdnWoCX88cet3FnZ2dnUU7-T8AAAAA@giganews.com>
Message-ID: <o1s2um$sa9$1@blaine.gmane.org>

On 2016-12-02, Wildman via Python-list <python-list at python.org> wrote:
> On Wed, 30 Nov 2016 14:39:02 +0200, Anssi Saari wrote:
>
>> There'll be a couple more issues with the printing but they should be
>> easy enough.
>
> I finally figured it out, I think.  I'm not sure if my changes are
> what you had in mind but it is working.  Below is the updated code.
> Thank you for not giving me the answer.  I was a good learning
> experience for me and that was my purpose in the first place.
>
> def format_ip(addr):
>     return str(int(addr[0])) + '.' + \ # replace ord() with int()
>            str(int(addr[1])) + '.' + \
>            str(int(addr[2])) + '.' + \
>            str(int(addr[3]))

This is a little more "pythonic":

    def format_ip(addr):
        return '.'.join(str(int(a) for a in addr))

I don't know what the "addr" array contains, but if addr is a byte
string, then the "int()" call is not needed, in Pythong 3, a byte is
already an integer:

    def format_ip(a):
       return '.'.join(str(b) for b in a)

addr = b'\x12\x34\x56\x78'

print(format_ip(addr))

If <addr> is an array of strings containing base-10 representations of
integers, then the str() call isn't needed either:

def format_ip(a):
    return '.'.join(s for s in a)

addr = ['12','34','56','78']

print(format_ip(addr))

-- 
Grant Edwards               grant.b.edwards        Yow! TAILFINS!! ... click
                                  at               ...
                              gmail.com            



From grant.b.edwards at gmail.com  Fri Dec  2 10:22:35 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 2 Dec 2016 15:22:35 +0000 (UTC)
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
 <5840fcf3$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o1s3jr$3vu$1@blaine.gmane.org>

On 2016-12-02, Steve D'Aprano <steve+python at pearwood.info> wrote:

> I'm not an expert on the low-level hardware details, so I welcome
> correction, but I think that you can probably expect that the OS can
> interrupt code execution between any two CPU instructions.

Yep, mostly.  Some CPUs have "lock" features that allow two or more
adjacent instructions to be atomic.  Such a "lock" feature is usually
used to implement atomic read-modify write operations (e.g. increment
a memory byte/word, set/clear a bit in a memory byte/word) on CPUs
that don't have any read-modify-write instructions.  In general CISC
processors like x86, AMD64, 68K have read-modify-write instructions
that allow you to increment a memory location or set/clear a bit in
memory with a single instruction:

    INC.W  [R0]    # increment memory word whose addr is in register R0

Many RISC CPUs don't (many ARMs) and require three instructions to
increment a word in memory:

    LD  R1,[R0]    # read into register R1 memory word whose addr is in R0
    INC R1         # increment value in R1
    ST  R1,[R0]    # store from regster R1 into memory at addr in R0

Some such RISC CPUs have a mechism (other than the normal interrupt
enable/disable mechanism) to allow you to lock the CPU for the
duration of those three instructions to ensure that they are atomic.

-- 
Grant Edwards               grant.b.edwards        Yow! I request a weekend in
                                  at               Havana with Phil Silvers!
                              gmail.com            



From best_lay at yahoo.com  Fri Dec  2 12:11:56 2016
From: best_lay at yahoo.com (Wildman)
Date: Fri, 02 Dec 2016 11:11:56 -0600
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <vg3wpfl9m6x.fsf@coffee.modeemi.fi>
 <mailman.9.1480509554.2387.python-list@python.org>
 <EPmdnWoCX88cet3FnZ2dnUU7-T8AAAAA@giganews.com>
 <o1s2um$sa9$1@blaine.gmane.org>
 <mailman.6.1480691492.2253.python-list@python.org>
Message-ID: <oeudnRnI3arBNtzFnZ2dnUU7-Y2dnZ2d@giganews.com>

On Fri, 02 Dec 2016 15:11:18 +0000, Grant Edwards wrote:

> I don't know what the "addr" array contains, but if addr is a byte
> string, then the "int()" call is not needed, in Pythong 3, a byte is
> already an integer:
> 
>     def format_ip(a):
>        return '.'.join(str(b) for b in a)
> 
> addr = b'\x12\x34\x56\x78'
> 
> print(format_ip(addr))

It is a byte string just like your 'addr =' example and
the above code works perfectly.  Thank you.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From torriem at gmail.com  Fri Dec  2 12:32:57 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 2 Dec 2016 10:32:57 -0700
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <b54fe8a8-e6e2-4d18-8031-534aefa0fd84@googlegroups.com>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
 <b54fe8a8-e6e2-4d18-8031-534aefa0fd84@googlegroups.com>
Message-ID: <d1997a7d-f171-b15f-2669-f9a781a81b71@gmail.com>

On 12/01/2016 08:39 PM, Ned Batchelder wrote:
> On Thursday, December 1, 2016 at 7:26:18 PM UTC-5, DFS wrote:
>> How is it possible that the 'if' portion runs, then 44/100,000ths of a 
>> second later my process yields to another process which deletes the 
>> file, then my process continues.
> 
> A modern computer is running dozens or hundreds (or thousands!) of
> processes "all at once". How they are actually interleaved on the
> small number of actual processors is completely unpredictable. There
> can be an arbitrary amount of time passing between any two processor
> instructions.

I'm not seeing DFS's original messages, so I'm assuming he's blocked on
the mailing list somehow (I have a vague memory of that).

But reading your quotes of his message reminds me that when I was a
system administrator, I saw this behavior all the time when using rsync
on a busy server.  It is very common for files to disappear out from
under rsync.  Often we just ignore this, but it can lead to incomplete
backups depending on what disappeared.  It was a great day when I moved
to using ZFS snapshots in conjunction with rsync to prevent this problem.

Anyway the point is to confirm what you're saying that yes this race
condition is a very real problem and actually happens in the real world.
 He's obviously thinking way too narrowly about the issue and neglecting
to consider the effects of multiple processes other than the running
python program.  It could even be something as simple as trying to read
a config program that might be opened in an editor by the user. Many
editors will move/rename the old file, then write the new version, and
then delete the old file.  If the reading of the file happens to be
attempted after the old file was renamed but before the new version was
written, the file is gone.  This has happened to me before, especially
if load average was high and the disk has fallen behind.


From marko at pacujo.net  Fri Dec  2 13:44:32 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 02 Dec 2016 20:44:32 +0200
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
 <5840fcf3$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o1s3jr$3vu$1@blaine.gmane.org>
 <mailman.7.1480692176.2253.python-list@python.org>
Message-ID: <871sxquq5r.fsf@elektro.pacujo.net>

Grant Edwards <grant.b.edwards at gmail.com>:
> In general CISC processors like x86, AMD64, 68K have read-modify-write
> instructions that allow you to increment a memory location or
> set/clear a bit in memory with a single instruction:
>
>     INC.W  [R0]    # increment memory word whose addr is in register R0

The x86 instruction set has a special lock prefix for the purpose:

   <URL: http://stackoverflow.com/questions/8891067/what-does-the-lock-ins
   truction-mean-in-x86-assembly>


Marko


From grant.b.edwards at gmail.com  Fri Dec  2 14:39:39 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 2 Dec 2016 19:39:39 +0000 (UTC)
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <vg3wpfl9m6x.fsf@coffee.modeemi.fi>
 <mailman.9.1480509554.2387.python-list@python.org>
 <EPmdnWoCX88cet3FnZ2dnUU7-T8AAAAA@giganews.com>
 <o1s2um$sa9$1@blaine.gmane.org>
 <mailman.6.1480691492.2253.python-list@python.org>
 <oeudnRnI3arBNtzFnZ2dnUU7-Y2dnZ2d@giganews.com>
Message-ID: <o1silr$oqj$1@blaine.gmane.org>

On 2016-12-02, Wildman via Python-list <python-list at python.org> wrote:
> On Fri, 02 Dec 2016 15:11:18 +0000, Grant Edwards wrote:
>
>> I don't know what the "addr" array contains, but if addr is a byte
>> string, then the "int()" call is not needed, in Pythong 3, a byte is
>> already an integer:
>> 
>>     def format_ip(a):
>>        return '.'.join(str(b) for b in a)
>> 
>> addr = b'\x12\x34\x56\x78'
>> 
>> print(format_ip(addr))
>
> It is a byte string just like your 'addr =' example and
> the above code works perfectly.

More importantly, you've now learned about generator comprehensions
(aka generator expressions) and the string type's "join" method.  ;)

-- 
Grant Edwards               grant.b.edwards        Yow! My Aunt MAUREEN was a
                                  at               military advisor to IKE &
                              gmail.com            TINA TURNER!!



From grant.b.edwards at gmail.com  Fri Dec  2 14:51:17 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 2 Dec 2016 19:51:17 +0000 (UTC)
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
 <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o1ptmr$cbv$3@dont-email.me>
 <cf28d967-6934-472b-aa74-7b663659d27b@googlegroups.com>
 <o1qf0h$bhu$1@dont-email.me>
 <5840fcf3$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o1s3jr$3vu$1@blaine.gmane.org>
 <mailman.7.1480692176.2253.python-list@python.org>
 <871sxquq5r.fsf@elektro.pacujo.net>
Message-ID: <o1sjbl$n6v$1@blaine.gmane.org>

On 2016-12-02, Marko Rauhamaa <marko at pacujo.net> wrote:
> Grant Edwards <grant.b.edwards at gmail.com>:
>> In general CISC processors like x86, AMD64, 68K have read-modify-write
>> instructions that allow you to increment a memory location or
>> set/clear a bit in memory with a single instruction:
>>
>>     INC.W  [R0]    # increment memory word whose addr is in register R0
>
> The x86 instruction set has a special lock prefix for the purpose:
>
>    <URL: http://stackoverflow.com/questions/8891067/what-does-the-lock-instruction-mean-in-x86-assembly>

The x86 already has single-instruction read-modify-write instruction,
so there's no possibility of your task being interrupted/suspended
during those single-instruction operations (which was sort of the
original topic).  What the lock prefix does is lock the _bus_ for the
duration of that one instruction so that other bus masters (other CPU
cores or DMA masters) can't access the memory bus in the middle of the
R-M-W instruction.  Obiously, if you've got multiple bus masters,
merely locking the CPU and not the bus may not be sufficient to avoid
race conditions.  Locking the CPU only prevents races between
different execution contexts (processes, threads, interrupt handlers)
on that one CPU.  If you've only got one CPU, and you know that none
of the DMA masters are going to write to your memory location, then
you don't need to lock the bus as long as your operation is a single
instruction.

-- 
Grant Edwards               grant.b.edwards        Yow! Here I am at the flea
                                  at               market but nobody is buying
                              gmail.com            my urine sample bottles ...



From codewizard at gmail.com  Fri Dec  2 15:50:57 2016
From: codewizard at gmail.com (codewizard at gmail.com)
Date: Fri, 2 Dec 2016 12:50:57 -0800 (PST)
Subject: What do you think: good idea to launch a marketplace on
 python+django?
In-Reply-To: <9ee77ca5-f2c8-48ef-bae8-bc4a1feef483@googlegroups.com>
References: <9ee77ca5-f2c8-48ef-bae8-bc4a1feef483@googlegroups.com>
Message-ID: <70143d20-b528-4c14-8ec9-09ebaf108629@googlegroups.com>

On Friday, December 2, 2016 at 2:01:57 AM UTC-5, Gus_G wrote:
> Hello, what do you think about building a marketplace website on connection of python+django? End effect-side should look and work similar to these: https://zoptamo.com/uk/s-abs-c-uk, https://www.ownerdirect.com/ . What are your opinions on this idea? Maybe there is other, better way to build it?

Something like this: https://marketplace.django-cms.org/en/ ?


From jobmattcon at gmail.com  Fri Dec  2 19:01:49 2016
From: jobmattcon at gmail.com (Ho Yeung Lee)
Date: Fri, 2 Dec 2016 16:01:49 -0800 (PST)
Subject: compile error when using override
In-Reply-To: <5841004e$0$1619$c3e8da3$5496439d@news.astraweb.com>
References: <07416672-be69-4329-a3de-d198eb831453@googlegroups.com>
 <58400753$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <48fc3a2a-4cae-4e03-bc1a-c23151ec3ff2@googlegroups.com>
 <5841004e$0$1619$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <844da8c2-f161-414d-b8c9-2b5acd66476d@googlegroups.com>

from __future__ import division 
from sympy import * 
x, y, z, t = symbols('x y z t') 
k, m, n = symbols('k m n', integer=True) 
f, g, h = symbols('f g h', cls=Function) 

class AA(object): 
    @staticmethod 
    def __additionFunction__(a1, a2): 
        return a1*a2 #Put what you want instead of this 
    def __multiplyFunction__(a1, a2): 
        return a1*a2+a1 #Put what you want instead of this 
    def __divideFunction__(a1, a2): 
        return a1*a1*a2 #Put what you want instead of this 
    def __init__(self, value): 
        self.value = value 
    def __add__(self, other): 
        return self.value*other.value 
    def __mul__(self, other): 
        return self.value*other.value + other.value 
    def __div__(self, other): 
        return self.value*other.value*other.value 

solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y) 

>>> solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'Add' and 'AA'


still error, 

actually i invented 3 valued logic algebraic operation which is quintessential and would replace into it if this succeed


On Friday, December 2, 2016 at 1:02:19 PM UTC+8, Steve D'Aprano wrote:
> On Fri, 2 Dec 2016 01:35 pm, Ho Yeung Lee wrote:
> 
> > from __future__ import division
> > import ast
> > from sympy import *
> > x, y, z, t = symbols('x y z t')
> > k, m, n = symbols('k m n', integer=True)
> > f, g, h = symbols('f g h', cls=Function)
> > import inspect
> 
> Neither ast nor inspect is used. Why import them?
> 
> The only symbols you are using are x and y.
> 
> 
> > def op2(a,b):
> >     return a*b+a
> 
> This doesn't seem to be used. Get rid of it.
> 
> 
> > class AA(object):
> >     @staticmethod
> >     def __additionFunction__(a1, a2):
> >         return a1*a2 #Put what you want instead of this
> >     def __multiplyFunction__(a1, a2):
> >         return a1*a2+a1 #Put what you want instead of this
> >     def __divideFunction__(a1, a2):
> >         return a1*a1*a2 #Put what you want instead of this
> 
> None of those methods are used. Get rid of them.
> 
> >     def __init__(self, value):
> >         self.value = value
> >     def __add__(self, other):
> >         return self.value*other.value
> 
> Sorry, you want AA(5) + AA(2) to return 10?
> 
> >     def __mul__(self, other):
> >         return self.value*other.value + other.value
> >     def __div__(self, other):
> >         return self.value*other.value*other.value
> > 
> > solve([AA(x)*AA(y) + AA(-1), AA(x) + AA(-2)], x, y)
> 
> I don't understand what you are trying to do here. What result are you
> execting?
> 
> Maybe you just want this?
> 
> from sympy import solve, symbols
> x, y = symbols('x y')
> print( solve([x*y - 1, x - 2], x, y) )
> 
> which prints the result:
> [(2, 1/2)]
> 
> 
> Perhaps if you explain what you are trying to do, we can help better.
> 
> But please, cut down your code to only code that is being used!
> 
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.



From best_lay at yahoo.com  Sat Dec  3 13:00:59 2016
From: best_lay at yahoo.com (Wildman)
Date: Sat, 03 Dec 2016 12:00:59 -0600
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
 <vg3wpfl9m6x.fsf@coffee.modeemi.fi>
 <mailman.9.1480509554.2387.python-list@python.org>
 <EPmdnWoCX88cet3FnZ2dnUU7-T8AAAAA@giganews.com>
 <o1s2um$sa9$1@blaine.gmane.org>
 <mailman.6.1480691492.2253.python-list@python.org>
 <oeudnRnI3arBNtzFnZ2dnUU7-Y2dnZ2d@giganews.com>
 <o1silr$oqj$1@blaine.gmane.org>
 <mailman.9.1480707595.2253.python-list@python.org>
Message-ID: <A4ydnduOIL7Gld7FnZ2dnUU7-WWdnZ2d@giganews.com>

On Fri, 02 Dec 2016 19:39:39 +0000, Grant Edwards wrote:

> On 2016-12-02, Wildman via Python-list <python-list at python.org> wrote:
>> On Fri, 02 Dec 2016 15:11:18 +0000, Grant Edwards wrote:
>>
>>> I don't know what the "addr" array contains, but if addr is a byte
>>> string, then the "int()" call is not needed, in Pythong 3, a byte is
>>> already an integer:
>>> 
>>>     def format_ip(a):
>>>        return '.'.join(str(b) for b in a)
>>> 
>>> addr = b'\x12\x34\x56\x78'
>>> 
>>> print(format_ip(addr))
>>
>> It is a byte string just like your 'addr =' example and
>> the above code works perfectly.
> 
> More importantly, you've now learned about generator comprehensions
> (aka generator expressions) and the string type's "join" method.  ;)

I have seen the join method before but because of my lack
of experience it didn't occur to me to use it.  I bet I
will remember it from now on.  I stuck a few code examples
into my 'snips' directory.

Generator expressions are new to me.  I have seen it's use
but I didn't understand what it was and what it was doing,
until now.  Thanks again.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From juan0christian at gmail.com  Sat Dec  3 17:14:09 2016
From: juan0christian at gmail.com (Juan C.)
Date: Sat, 3 Dec 2016 20:14:09 -0200
Subject: How to properly retrieve data using requests + bs4 from multiple
 pages in a site?
In-Reply-To: <CAAp0bGutnU=KGi9gCnZ42cb+FfdE4HVXgfuXddcRBhqsvWz9iA@mail.gmail.com>
References: <CAAp0bGutnU=KGi9gCnZ42cb+FfdE4HVXgfuXddcRBhqsvWz9iA@mail.gmail.com>
Message-ID: <CAAp0bGv7vO0eRudEOFEeEHqLuf2tiHgydN4AWZ3VOnJtYXt8=w@mail.gmail.com>

On Thu, Dec 1, 2016 at 10:07 PM, Juan C. <juan0christian at gmail.com> wrote:
> It works, but it has a big issue: it gets all data from all
units/courses/assignments at the same time, and this isn't very useful as I
don't care about data from units from 1-2 years ago. How can I change the
logic so it just gets the data I need at a given moment? For example, I may
need to dump data for an entire unit, or just one course, or maybe even
just one assignment. How can I achieve this behavior? Another "issue", I
feel like handing my 'session' that I instantiated at user.py to program,
then unit, then course and then assignment is a poor design, how can I make
it better?
>
> Any other suggestions are welcome.

Oh, forgot to tell, I'm using Python 3.5.2 x64.


From rxjwg98 at gmail.com  Sat Dec  3 18:08:52 2016
From: rxjwg98 at gmail.com (Robert)
Date: Sat, 3 Dec 2016 15:08:52 -0800 (PST)
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
Message-ID: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>

Hi,

I am trying to understand the meaning of the below code snippet. Though I have
a Python IDLE at computer, I can't get a way to know below line:

if k in [0, len(n_trials) - 1] else None

I feel it is strange for what returns when the 'if' condition is true?
The second part 'None' is clear to me though.

Could you explain it to me?


thanks,







%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials) - 1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True


From rxjwg98 at gmail.com  Sat Dec  3 18:11:38 2016
From: rxjwg98 at gmail.com (Robert)
Date: Sat, 3 Dec 2016 15:11:38 -0800 (PST)
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
Message-ID: <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>

On Saturday, December 3, 2016 at 6:09:02 PM UTC-5, Robert wrote:
> Hi,
> 
> I am trying to understand the meaning of the below code snippet. Though I have
> a Python IDLE at computer, I can't get a way to know below line:
> 
> if k in [0, len(n_trials) - 1] else None
> 
> I feel it is strange for what returns when the 'if' condition is true?
> The second part 'None' is clear to me though.
> 
> Could you explain it to me?
> 
> 
> thanks,
> 
> 
> 
> 
> 
> 
> 
> %matplotlib inline
> from IPython.core.pylabtools import figsize
> import numpy as np
> from matplotlib import pyplot as plt
> figsize(11, 9)
> 
> import scipy.stats as stats
> 
> dist = stats.beta
> n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
> data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
> x = np.linspace(0, 1, 100)
> 
> # For the already prepared, I'm using Binomial's conj. prior.
> for k, N in enumerate(n_trials):
>     sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
>     plt.xlabel("$p$, probability of heads") \
>         if k in [0, len(n_trials) - 1] else None
>     plt.setp(sx.get_yticklabels(), visible=False)
>     heads = data[:N].sum()
>     y = dist.pdf(x, 1 + heads, 1 + N - heads)
>     plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
>     plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
>     plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
> 
>     leg = plt.legend()
>     leg.get_frame().set_alpha(0.4)
>     plt.autoscale(tight=True

I just notice that there is a slash character (\) before the if line.
What is it for?

I've learn Python for a while, but I don't use it for more than 2 years now.
Thanks.


From rosuav at gmail.com  Sat Dec  3 18:27:42 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 4 Dec 2016 10:27:42 +1100
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
Message-ID: <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>

On Sun, Dec 4, 2016 at 10:11 AM, Robert <rxjwg98 at gmail.com> wrote:
> I just notice that there is a slash character (\) before the if line.
> What is it for?

Yes, that's important. The entire line of code is:

    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials) - 1] else None

The backslash means "this continues on the next line". The ternary
conditional looks like this:

5 if 1 < 2 else 7

Since 1 < 2, this has the value of 5. If not, it would have the value 7.

But the expression result isn't even used. So this is better written:

if k in [0, len(n_trials) - 1]:
   plt.xlabel("$p$, probability of heads")

ChrisA


From python at mrabarnett.plus.com  Sat Dec  3 18:51:49 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 3 Dec 2016 23:51:49 +0000
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
Message-ID: <cb0b4102-abf7-0881-643f-35315522a038@mrabarnett.plus.com>

On 2016-12-03 23:11, Robert wrote:
> On Saturday, December 3, 2016 at 6:09:02 PM UTC-5, Robert wrote:
>> Hi,
>>
>> I am trying to understand the meaning of the below code snippet. Though I have
>> a Python IDLE at computer, I can't get a way to know below line:
>>
>> if k in [0, len(n_trials) - 1] else None
>>
>> I feel it is strange for what returns when the 'if' condition is true?
>> The second part 'None' is clear to me though.
>>
>> Could you explain it to me?
>>
>>
>> thanks,
>>
>>
>> %matplotlib inline
>> from IPython.core.pylabtools import figsize
>> import numpy as np
>> from matplotlib import pyplot as plt
>> figsize(11, 9)
>>
>> import scipy.stats as stats
>>
>> dist = stats.beta
>> n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
>> data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
>> x = np.linspace(0, 1, 100)
>>
>> # For the already prepared, I'm using Binomial's conj. prior.
>> for k, N in enumerate(n_trials):
>>     sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
>>     plt.xlabel("$p$, probability of heads") \
>>         if k in [0, len(n_trials) - 1] else None
>>     plt.setp(sx.get_yticklabels(), visible=False)
>>     heads = data[:N].sum()
>>     y = dist.pdf(x, 1 + heads, 1 + N - heads)
>>     plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
>>     plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
>>     plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
>>
>>     leg = plt.legend()
>>     leg.get_frame().set_alpha(0.4)
>>     plt.autoscale(tight=True
>
> I just notice that there is a slash character (\) before the if line.
> What is it for?
>
> I've learn Python for a while, but I don't use it for more than 2 years now.
> Thanks.
>
The backslash at the end of the line indicates that that the statement 
continues onto the next line, so it's the same as:

     plt.xlabel("$p$, probability of heads") if k in [0, len(n_trials) - 
1] else None

However, that line is weird!

In Python there's a "ternary operator". The docs say:

"""The expression x if C else y first evaluates the condition, C rather 
than x. If C is true, x is evaluated and its value is returned; 
otherwise, y is evaluated and its value is returned."""

Suppose you have the expression:

     "even" if x % 2 == 0 else "odd"

If x is a multiple of 2, that expression will evaluate to "even", else 
it will evaluate to "odd".

The line in your code is misusing it as a statement. Normally you would 
write this instead:

     if k in [0, len(n_trials) - 1]:
         plt.xlabel("$p$, probability of heads")

Why was it written that way? I have no idea, it's just weird...



From tjreedy at udel.edu  Sat Dec  3 19:10:50 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 3 Dec 2016 19:10:50 -0500
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
 <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>
Message-ID: <o1vmu7$3i8$1@blaine.gmane.org>

On 12/3/2016 6:27 PM, Chris Angelico wrote:
> On Sun, Dec 4, 2016 at 10:11 AM, Robert <rxjwg98 at gmail.com> wrote:
>> I just notice that there is a slash character (\) before the if line.
>> What is it for?
>
> Yes, that's important. The entire line of code is:
>
>     plt.xlabel("$p$, probability of heads") \
>         if k in [0, len(n_trials) - 1] else None
>
> The backslash means "this continues on the next line".
 > The ternary conditional looks like this:

> 5 if 1 < 2 else 7
>
> Since 1 < 2, this has the value of 5. If not, it would have the value 7.
>
> But the expression result isn't even used. So this is better written:

matplotlib.pyplot.xlabel sets x-axis scaling, with no documented return 
value.
http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.xlabel
If, as seems reasonable to assume, it returns None, the value of the 
expression is 'None if x else None', which is to say, None.

> if k in [0, len(n_trials) - 1]:
>    plt.xlabel("$p$, probability of heads")

-- 
Terry Jan Reedy



From rosuav at gmail.com  Sat Dec  3 19:31:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 4 Dec 2016 11:31:46 +1100
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <o1vmu7$3i8$1@blaine.gmane.org>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
 <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>
 <o1vmu7$3i8$1@blaine.gmane.org>
Message-ID: <CAPTjJmqjQMQB9AS7H0H0tEGEnvN4MORB494rm0QhTaU1_uZScw@mail.gmail.com>

On Sun, Dec 4, 2016 at 11:10 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> But the expression result isn't even used. So this is better written:
>
>
> matplotlib.pyplot.xlabel sets x-axis scaling, with no documented return
> value.
> http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.xlabel
> If, as seems reasonable to assume, it returns None, the value of the
> expression is 'None if x else None', which is to say, None.

Hardly matters what the return value is, given that the code ignores it.

ChrisA


From benjamin at python.org  Sat Dec  3 23:18:41 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 03 Dec 2016 20:18:41 -0800
Subject: [RELEASE] Python 2.7.13 release candidate 1
Message-ID: <1480825121.973481.807578801.7C0355E5@webmail.messagingengine.com>

It is my pleasure to announce the first release candidate of Python
2.7.13, a new bugfix release in the Python 2.7x series.

Downloads may be found on python.org:
     https://www.python.org/downloads/release/python-2713rc1/

Please test the release and report any bugs to
    https://bugs.python.org

A final release is scheduled for 2 weeks time.

Servus,
Benjamin
(on behalf of all of 2.7's contributors)


From jobmattcon at gmail.com  Sun Dec  4 00:20:47 2016
From: jobmattcon at gmail.com (Ho Yeung Lee)
Date: Sat, 3 Dec 2016 21:20:47 -0800 (PST)
Subject: how to override the solver function in sympy?
Message-ID: <de00c53d-da38-449d-85be-33c3b9895d51@googlegroups.com>

how to override the solver function in sympy?


From hpj at urpla.net  Sun Dec  4 07:22:44 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Sun, 04 Dec 2016 13:22:44 +0100
Subject: distutils_ui 0.1.1 released
Message-ID: <18074248.yx9y0mqd1H@xrated>

For those of you, who like PyQt{4,5} as much as I do, as well as for those who 
don't like it that much, because of the poor integration with setuptools 
et.al., here's another piece of software to bridge the gap:

	A distutils build extension for PyQt{4,5} applications

that makes handling the PyQt tool chain easier than ever:

	https://pypi.python.org/pypi/distutils_ui

Ahem, well, it wasn't that easy before. Most of us were using dreaded 
Makefiles or other such crutches to generate translation files, .py modules of 
forms, and resource modules. Scratch the crutches, here's what you're looking 
for.

Feedback welcome.

Enjoy,
Pete




From tjreedy at udel.edu  Sun Dec  4 14:14:14 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 4 Dec 2016 14:14:14 -0500
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <CAPTjJmqjQMQB9AS7H0H0tEGEnvN4MORB494rm0QhTaU1_uZScw@mail.gmail.com>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
 <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>
 <o1vmu7$3i8$1@blaine.gmane.org>
 <CAPTjJmqjQMQB9AS7H0H0tEGEnvN4MORB494rm0QhTaU1_uZScw@mail.gmail.com>
Message-ID: <o21pu4$p8g$1@blaine.gmane.org>

On 12/3/2016 7:31 PM, Chris Angelico wrote:
> On Sun, Dec 4, 2016 at 11:10 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>>> But the expression result isn't even used. So this is better written:
>>
>> matplotlib.pyplot.xlabel sets x-axis scaling, with no documented return
>> value.
>> http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.xlabel
>> If, as seems reasonable to assume, it returns None, the value of the
>> expression is 'None if x else None', which is to say, None.
>
> Hardly matters what the return value is, given that the code ignores it.

I was not clear enough.  To me, a 'conditional expression' that has an 
unconditional value is a bit confusing.  So I think the conditional 
side-effect would be better written the way you did regardless of the 
use or not of the value.  If the unconditional value were not ignored, I 
think it would be better used by itself.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Sun Dec  4 14:36:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 5 Dec 2016 06:36:58 +1100
Subject: What meaning is "if k in [0, len(n_trials) - 1] else None"?
In-Reply-To: <o21pu4$p8g$1@blaine.gmane.org>
References: <d1274140-bf0b-4f03-86a9-90c953d1c41f@googlegroups.com>
 <548abc5e-c532-4327-a55d-79050d83029b@googlegroups.com>
 <CAPTjJmr6Yoe9J+f47OpHZvmBPGPbFh37M=yjj2JdF0X0omoYFQ@mail.gmail.com>
 <o1vmu7$3i8$1@blaine.gmane.org>
 <CAPTjJmqjQMQB9AS7H0H0tEGEnvN4MORB494rm0QhTaU1_uZScw@mail.gmail.com>
 <o21pu4$p8g$1@blaine.gmane.org>
Message-ID: <CAPTjJmqvxLyWXt1CjgbGRgpcFWiFCehyy=qKGtd_45C-H1=-wQ@mail.gmail.com>

On Mon, Dec 5, 2016 at 6:14 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 12/3/2016 7:31 PM, Chris Angelico wrote:
>>
>> On Sun, Dec 4, 2016 at 11:10 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>>>>
>>>> But the expression result isn't even used. So this is better written:
>>>
>>>
>>> matplotlib.pyplot.xlabel sets x-axis scaling, with no documented return
>>> value.
>>> http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.xlabel
>>> If, as seems reasonable to assume, it returns None, the value of the
>>> expression is 'None if x else None', which is to say, None.
>>
>>
>> Hardly matters what the return value is, given that the code ignores it.
>
>
> I was not clear enough.  To me, a 'conditional expression' that has an
> unconditional value is a bit confusing.  So I think the conditional
> side-effect would be better written the way you did regardless of the use or
> not of the value.  If the unconditional value were not ignored, I think it
> would be better used by itself.

Fair enough. It's doubly bizarre - the expression result is constant
AND it's an expression being used as a statement that has an
equivalent statement form. Plus, the statement form takes two lines...
but the expression is over two lines anyway.

ChrisA


From bc at freeuk.com  Sun Dec  4 17:19:56 2016
From: bc at freeuk.com (BartC)
Date: Sun, 4 Dec 2016 22:19:56 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o21u2p$co1$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me>
Message-ID: <o224o1$5d2$1@dont-email.me>

On 04/12/2016 20:26, DFS wrote:
> $python program.py column1=2174 and column2='R'
>
>
> Windows (correct)
> $print sys.argv[3]
> column2='R'
>
> Linux (incorrect)
> $print sys.argv[3]
> column2=R
>
> It drops the apostrophes, and the subsequent db call throws an error:
> sqlite3.OperationalError: no such column: R
>
> The way it's used in code is:
> argcnt   = len(sys.argv)
> querystr = ' '.join(sys.argv[1:argcnt])
>
>
> I tried using dbl-quotes in the command line, and with the join()
> statement, and neither worked.
>
>
> Edit: I got it to work this way:
> column2="'R'"
>
> but that's bogus, and I don't want users to have to do that.

You can put double quotes around the whole thing:

  "column2='R'"

otherwise I don't know what the solution other than for a program be 
aware of the possibility and allow for either input, if there are no 
conflicts (for example if both R and 'R' are valid inputs and mean 
different things).

Command parameters /do/ behave differently between Windows and Linux, 
for example try writing *.* as that third parameter.

In Windows, it will print *.*.

In Linux, if you have 273 files in the current directory, if will print 
the name of the first, and there will be /272 further command 
parameters/, each the name of a file. (I couldn't believe this when I 
found out; one of my directories recently had 3.4 million files in it, I 
don't really want *.* expanded to 3.4m arguments. Here, the fix is again 
to use double quotes: "*.*". But what if the user doesn't do that?)

-- 
Bartc




From steve+python at pearwood.info  Sun Dec  4 17:52:03 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 05 Dec 2016 09:52:03 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me>
Message-ID: <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Mon, 5 Dec 2016 07:26 am, DFS wrote:

> $python program.py column1=2174 and column2='R'

Here is a simple script demonstrating the issue:


# --- program.py ---
import sys
print "argv:", sys.argv
print ' '.join(sys.argv[1:])



I haven't tested it on Windows, but on Linux it behaves as you describe (and
as Linux users will expect):

[steve at ando ~]$ python program.py column1=2174 and column2='R'
argv: ['program.py', 'column1=2174', 'and', 'column2=R']
column1=2174 and column2=R



This is *absolutely normal behaviour* for the most common shell on Linux,
bash. I would expect that the other common shells will do the same thing.
Quotation marks need to be escaped if you want the shell to pass them
through to the program, either like this:

column2="'R'"


or like this:

column2=\'R\'



> It drops the apostrophes, and the subsequent db call throws an error:
> sqlite3.OperationalError: no such column: R

I'm not sure how to interpret this error, so I'm guessing. Please correct me
if I'm wrong, but doesn't this mean that your column is called:

single quote R single quote

that is, literally 'R', which means that if you were using it in Python
code, you would have to write the column name as this?

"'R'"

If so, perhaps the best solution is to get rid of the quotes in the column
names, so that all your users, Windows and Linux, just write this:

program.py column1=2174 and column2=R


(P.S. what happens if they write 

program.py column1 = 2174 and column2 = R

instead?)



> The way it's used in code is:
> argcnt   = len(sys.argv)
> querystr = ' '.join(sys.argv[1:argcnt])

You can simplify that to:

querystr = ' '.join(sys.argv[1:])




> I tried using dbl-quotes in the command line, and with the join()
> statement, and neither worked.

By the time you join the arguments, its too late. You have to quote them
first.


> Edit: I got it to work this way:
> column2="'R'"
> 
> but that's bogus, and I don't want users to have to do that.

(1) It's not bogus.

(2) Linux users will expect that you have to escape quotation marks if you
want to pass them through the shell.

(3) If my interpretation is correct, what *is* bogus is that that your
column names include quotes in them. Get rid of the quotes, and your
problem goes away.

If that's not the case, then you'll either have to get your users to escape
the quotes, or you'll have to add them in yourself.

(In case this is not obvious by now, this is not a Python issue. This is
entirely due to the behaviour of the shell.)




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sun Dec  4 17:58:27 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 5 Dec 2016 09:58:27 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o224o1$5d2$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
Message-ID: <CAPTjJmpDdZEHvg+bV2UkmLqT765orOX4nLG_WGkkd1Rp+8AWZg@mail.gmail.com>

On Mon, Dec 5, 2016 at 9:19 AM, BartC <bc at freeuk.com> wrote:
> Command parameters /do/ behave differently between Windows and Linux, for
> example try writing *.* as that third parameter.
>
> In Windows, it will print *.*.
>
> In Linux, if you have 273 files in the current directory, if will print the
> name of the first, and there will be /272 further command parameters/, each
> the name of a file. (I couldn't believe this when I found out; one of my
> directories recently had 3.4 million files in it, I don't really want *.*
> expanded to 3.4m arguments. Here, the fix is again to use double quotes:
> "*.*". But what if the user doesn't do that?)

Technically this isn't a Win/Lin difference, but a shell difference.
The behaviours you're referring to are common to many Unix shells
(including bash, the most commonly used shell on typical Linux
systems). If you want a glob to be processed by the application,
rather than the shell, you have to escape it with quotes or
backslashes. Most of the time, it's easiest to write your app to
accept multiple file names, and let the shell expand them - there's a
lot more flexibility than just * and ?, and every program behaves the
same way, because it's the same shell parsing them all.

ChrisA


From greg.ewing at canterbury.ac.nz  Sun Dec  4 18:06:41 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 05 Dec 2016 12:06:41 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me>
 <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <eajls3FbnetU1@mid.individual.net>

Steve D'Aprano wrote:
> On Mon, 5 Dec 2016 07:26 am, DFS wrote:
> 
>> no such column: R
> 
> doesn't this mean that your column is called:
> 
> single quote R single quote

I think he intends it to be an SQL string literal (which uses
single quotes), but since the quotes disappeared, SQL is trying
to interpret it as a column name.

>>Edit: I got it to work this way:
>>column2="'R'"
>>
>>but that's bogus, and I don't want users to have to do that.

DFS: Is the argument always going to be a literal, or could
the user sometimes want to pass a column name?

If it's always a literal, your program could add the quotes
before passing it to SQL.

If it's not always a literal, you'll just have to deal with
the fact that the unix shell is a programming language that
has its own interpretation of quotes. Either the user will
have to escape the quotes, or you'll have to provide another
way of distinguishing literals from column names.

-- 
Greg


From steve+python at pearwood.info  Sun Dec  4 18:25:11 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 05 Dec 2016 10:25:11 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
Message-ID: <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>

On Mon, 5 Dec 2016 09:19 am, BartC wrote:

> Command parameters /do/ behave differently between Windows and Linux,
> for example try writing *.* as that third parameter.
> 
> In Windows, it will print *.*.
> 
> In Linux, if you have 273 files in the current directory, if will print
> the name of the first, and there will be /272 further command
> parameters/, each the name of a file. (I couldn't believe this when I
> found out; one of my directories recently had 3.4 million files in it, I
> don't really want *.* expanded to 3.4m arguments. Here, the fix is again
> to use double quotes: "*.*". But what if the user doesn't do that?)

If the user doesn't escape the wildcard, then the shell will expand it,
exactly as the user would expect.

I'm not sure why you were surprised by that. * is a shell wildcard. By using
a * you are explicitly telling the shell to expand it to any file that
matches. Did you think it was a regular character like 'a' and 'z'?

I think it boils down to what the user expects. Linux and Unix users tend to
be technically-minded folks who use the command line a lot and demand
powerful tools, and they expect that wildcards like * should be expanded.
Windows treats the command line as an afterthought, and until a few years
ago you were limited to a DOS shell. Today, your options are not as
limited: there's Powershell, and bash for Windows.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sun Dec  4 18:26:55 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 5 Dec 2016 10:26:55 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me>
 <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmqRD0dtf03dBtmf=Z42c=FGpguy1=BVQ9Van4yGjtzuZA@mail.gmail.com>

On Mon, Dec 5, 2016 at 9:52 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> I'm not sure how to interpret this error, so I'm guessing. Please correct me
> if I'm wrong, but doesn't this mean that your column is called:
>
> single quote R single quote
>
> that is, literally 'R', which means that if you were using it in Python
> code, you would have to write the column name as this?
>
> "'R'"
>

AIUI this is meant to be a string literal, which in SQL is surrounded
by single quotes. This also means that anyone who's using this script
needs to be comfortable with writing raw SQL; plus, there's no
protection against SQL injection, so anyone using the script has to
have full database permission. The best solution might well be to
change the protocol somewhat: instead of taking raw SQL on the command
line, take "column=value", parse that in Python, and provide the value
as a string (or maybe as "int if all digits else string").

ChrisA


From python at mrabarnett.plus.com  Sun Dec  4 18:48:33 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 4 Dec 2016 23:48:33 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me>
 <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <35543074-5176-7878-6a9a-90d39aaca5b6@mrabarnett.plus.com>

On 2016-12-04 22:52, Steve D'Aprano wrote:
> On Mon, 5 Dec 2016 07:26 am, DFS wrote:
>
>> $python program.py column1=2174 and column2='R'
>
> Here is a simple script demonstrating the issue:
>
>
> # --- program.py ---
> import sys
> print "argv:", sys.argv
> print ' '.join(sys.argv[1:])
>
>
>
> I haven't tested it on Windows, but on Linux it behaves as you describe (and
> as Linux users will expect):
>
> [steve at ando ~]$ python program.py column1=2174 and column2='R'
> argv: ['program.py', 'column1=2174', 'and', 'column2=R']
> column1=2174 and column2=R
>
[snip]

On Windows:

     py.exe program.py column1=2174 and column2='R'

gives:

     argv: ['program.py', 'column1=2174', 'and', "column2='R'"]
     column1=2174 and column2='R'

and:

     py.exe program.py column1=2174 and column2="R"

gives:

     argv: ['program.py', 'column1=2174', 'and', 'column2=R']
    column1=2174 and column2=R



From eryksun at gmail.com  Sun Dec  4 19:09:07 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 5 Dec 2016 00:09:07 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o224o1$5d2$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
Message-ID: <CACL+1atPM0Y2o=090pMD9qzL8gKaMyVDneWae7wmELfiKeciaQ@mail.gmail.com>

On Sun, Dec 4, 2016 at 10:19 PM, BartC <bc at freeuk.com> wrote:
>
> Command parameters /do/ behave differently between Windows and Linux, for
> example try writing *.* as that third parameter.
>
> In Windows, it will print *.*.

In Windows each program parses its own command line. Most C/C++
programs use the CRT's default [w]argv parsing. The CRT defaults to
disabling wildcard expansion. Link with [w]setargv.obj to enable this
feature:

https://msdn.microsoft.com/en-us/library/8bch7bkk

The CRT [w]argv parsing doesn't care about single quotes, but literal
double quotes need to be escaped.


From steve+comp.lang.python at pearwood.info  Mon Dec  5 00:20:28 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 05 Dec 2016 16:20:28 +1100
Subject: Agile
Message-ID: <5844f91f$0$1500$c3e8da3$5496439d@news.astraweb.com>

And you thought Agile development was simple:

https://pbs.twimg.com/media/CyjbCtGXcAEhQi8.jpg:large




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From rosuav at gmail.com  Mon Dec  5 00:36:57 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 5 Dec 2016 16:36:57 +1100
Subject: Agile
In-Reply-To: <5844f91f$0$1500$c3e8da3$5496439d@news.astraweb.com>
References: <5844f91f$0$1500$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmp56PjqR=p-CcZm1vxu1VRB5cm0hgaQtcQ2vFBW0FQtWg@mail.gmail.com>

On Mon, Dec 5, 2016 at 4:20 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> And you thought Agile development was simple:
>
> https://pbs.twimg.com/media/CyjbCtGXcAEhQi8.jpg:large
>

Agile may be complicated to explain, but it's so easy to use. This guy
understands it.

https://twitter.com/PHP_CEO/status/664237197365796864

ChrisA


From vek.m1234 at gmail.com  Mon Dec  5 00:43:27 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Mon, 05 Dec 2016 11:13:27 +0530
Subject: variable argument unpacking
References: <436682ac-ec56-4286-a36e-d5697a9fd83c@googlegroups.com>
 <nhunfi$d4r$1@ger.gmane.org>
 <mailman.15.1464000845.20402.python-list@python.org>
Message-ID: <o22unj$3r6$1@dont-email.me>

Peter Otten wrote:

> Mehrzad Irani wrote:
> 
>> Hi All,
>> 
>> Consider the situation
>> [cti at iranim-rhel python_cti]$ cat a.py
>> def a(a = 1, b = 2, c = 3, *d, **e):
>>         print(a, b, c)
>>         print(d)
>>         print(e)
>> 
>> r = {'e': 7, 'f': 8, 'g': 9}
>> 
>> 
>> a(**r)
>> a(3, **r)
>> 
>> r1 = (4,5,6)
>> 
>> a(3,2,1,*r1, **r)
>> a(*r1, **r)
>> 
>> r1 = (4,5,6,7)
>> a(*r1, **r)
>> 
>> [cti at iranim-rhel python_cti]$
>> 
>> The output for this program is as follows:
>> [cti at iranim-rhel python_cti]$ python a.py
>> (1, 2, 3)
>> ()
>> {'e': 7, 'g': 9, 'f': 8}
>> ------------------
>> (3, 2, 3)
>> ()
>> {'e': 7, 'g': 9, 'f': 8}
>> ------------------
>> (3, 2, 1)
>> (4, 5, 6)
>> {'e': 7, 'g': 9, 'f': 8}
>> ------------------
>> (4, 5, 6)
>> ()
>> {'e': 7, 'g': 9, 'f': 8}
>> ------------------
>> (4, 5, 6)
>> (7,)
>> {'e': 7, 'g': 9, 'f': 8}
>> 
>> This program shows, that for d to get assigned, I would need to first
>> assign a, b, c even though their default parameters have been set.
>> 
>> Also, if I would like to unpack a, b, c using e; I would get a
>> multiple assignment TypeError.
>> 
>> Therefore, my question is - is there a way to assign d, without going
>> through the assignments of a, b, c again, since they have already
>> been assigned defaults? (I think I am missing something simple here)
>> 
>> Thanks in advance.
> 
> Python 3 allows
> 
> $ cat b.py
> def a(*d, a=1, b=2, c=3, **e):
>         print(a, b, c)
>         print(d)
>         print(e)
> 
> r = {'e': 7, 'f': 8, 'g': 9}
> 
> 
> a(**r)
> a(3, **r)
> 
> r1 = (4,5,6)
> 
> a(3,2,1,*r1, **r)
> a(*r1, **r)
> 
> r1 = (4,5,6,7)
> a(*r1, **r)
> $ python3 b.py
> 1 2 3
> ()
> {'e': 7, 'f': 8, 'g': 9}
> 1 2 3
> (3,)
> {'e': 7, 'f': 8, 'g': 9}
> 1 2 3
> (3, 2, 1, 4, 5, 6)
> {'e': 7, 'f': 8, 'g': 9}
> 1 2 3
> (4, 5, 6)
> {'e': 7, 'f': 8, 'g': 9}
> 1 2 3
> (4, 5, 6, 7)
> {'e': 7, 'f': 8, 'g': 9}
> 
> Perhaps that is more to your liking? I find the output as unreadable
> as of your a.py, so I won't bother to check...

import inspect, linecache

def a(a = 1, b = 2, c = 3, l = 0, *d, **e):
        lineno = inspect.currentframe().f_back.f_lineno
        print 'line', lineno, linecache.getline(__file__, lineno),\
            linecache.getline(__file__, lineno-1)
        print(a, b, c)
        print(d)
        print(e)
        print '----------------'

r = {'e': 7, 'f': 8, 'g': 9}
a(**r)

a(3, **r)

r1 = (4,5,6)
a(3,2,1,*r1, **r)
a(*r1, **r)

r1 = (4,5,6,7)
a(*r1, **r)

(thanks to erica on freenode who linked me to: 
http://code.activestate.com/recipes/145297-grabbing-the-current-line-number-easily/)


From magdalena.guz80 at gmail.com  Mon Dec  5 01:14:49 2016
From: magdalena.guz80 at gmail.com (Gus_G)
Date: Sun, 4 Dec 2016 22:14:49 -0800 (PST)
Subject: What do you think: good idea to launch a marketplace on
 python+django?
In-Reply-To: <70143d20-b528-4c14-8ec9-09ebaf108629@googlegroups.com>
References: <9ee77ca5-f2c8-48ef-bae8-bc4a1feef483@googlegroups.com>
 <70143d20-b528-4c14-8ec9-09ebaf108629@googlegroups.com>
Message-ID: <0647a698-9823-4d5a-a9eb-adeece741a45@googlegroups.com>

W dniu pi?tek, 2 grudnia 2016 21:51:08 UTC+1 u?ytkownik codew... at gmail.com napisa?:
 
> Something like this: https://marketplace.django-cms.org/en/ ?

Hey, that's nice solution, good to know that there is site like this :D Thanks


From steve+comp.lang.python at pearwood.info  Mon Dec  5 03:33:09 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 05 Dec 2016 19:33:09 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me>
 <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o230sk$8sa$1@dont-email.me>
Message-ID: <58452645$0$1621$c3e8da3$5496439d@news.astraweb.com>

On Monday 05 December 2016 17:20, DFS wrote:

>>> Edit: I got it to work this way:
>>> column2="'R'"
>>>
>>> but that's bogus, and I don't want users to have to do that.
>>
>> (1) It's not bogus.
> 
> 
> It's extremely bogus.  It's discarding part of my input.

When you type 

name = 'Fred'

do you expect that the first character of the variable name is a quotation 
mark? No? Then why is it okay for Python to discard the quotation mark but not 
bash or some other shell?

Shells don't just repeat the characters you type, they interpret them. 
Characters like $ & * ? [ ] { } and others have special meaning to the shell, 
as do quotation marks.


[...]
>> (3) If my interpretation is correct, what *is* bogus is that that your
>> column names include quotes in them.
> 
> 
> Of course not.  I'm not sure I've ever seen quotes or apostrophes in a
> column name.

*shrug* It would be a pretty crazy thing to do, and I'm very glad to hear that 
my wild guess as to what was going on was wrong.

> And that's a pretty crazy interpretation that nothing in my post would
> lead you to.

That's the curse of knowledge speaking: you're familiar enough with SQL that 
you can no longer remember what it was like to be unfamiliar with it.

I'm not familiar with SQL syntax, and to me, the fact that SQL was complaining 
when the quotation marks were missing from the column name suggested to me that 
the quotes were part of the name. To me, the analogy that came to my mind was 
the similar-looking error in the shell:


steve at runes:~$ ls "'foo'"
'foo'
steve at runes:~$ ls 'foo'
ls: cannot access foo: No such file or directory

The quotation marks are part of the filename, and so they need to be protected 
from the shell or else you get an error quite similar to the one you got:

    no such column: R


but there is (or so it appeared) a column 'R' (hence it looks like the quotes 
are part of the column name).

Anyway, I'm glad you have a solution that satisfies you.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From bc at freeuk.com  Mon Dec  5 06:42:08 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 11:42:08 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o23jo4$30k$1@dont-email.me>

On 04/12/2016 23:25, Steve D'Aprano wrote:
> On Mon, 5 Dec 2016 09:19 am, BartC wrote:
>
>> Command parameters /do/ behave differently between Windows and Linux,
>> for example try writing *.* as that third parameter.
>>
>> In Windows, it will print *.*.
>>
>> In Linux, if you have 273 files in the current directory, if will print
>> the name of the first, and there will be /272 further command
>> parameters/, each the name of a file. (I couldn't believe this when I
>> found out; one of my directories recently had 3.4 million files in it, I
>> don't really want *.* expanded to 3.4m arguments. Here, the fix is again
>> to use double quotes: "*.*". But what if the user doesn't do that?)
>
> If the user doesn't escape the wildcard, then the shell will expand it,
> exactly as the user would expect.
>
> I'm not sure why you were surprised by that. * is a shell wildcard. By using
> a * you are explicitly telling the shell to expand it to any file that
> matches.

I don't know what a shell is. To me, it's some sort of user interface to 
the OS. So if someone types:

   > X A B C

You would expect X to be launched, and be given arguments A, B and C. 
You wouldn't expect any of them to be expanded to some unknown number of 
arguments.

In the same way that in code, you don't expect X(A,B,C) to be expanded 
to X(A,B0,B1,B2,B3,B4,B5,....., C) if B happens to specify a slice.


> Did you think it was a regular character like 'a' and 'z'?

If one of the parameters was a regular expression, would you expect it 
to be expanded to the entire possible set of inputs that match the 
expression?

> I think it boils down to what the user expects. Linux and Unix users tend to
> be technically-minded folks who use the command line a lot and demand
> powerful tools, and they expect that wildcards like * should be expanded.

Which is dumb. How does the shell know exactly what I want to do with 
*.* or f*.c or whatever? Perhaps I don't want the expanded list right 
now (all the 3.4 million elements); perhaps I want to apply the same 
pattern to several directories; perhaps I'm passing it on to another 
program; perhaps I'm going to be writing it as another script; perhaps I 
just want to print out the parameter list; perhaps I want to transform 
or validate the pattern first; maybe I need to verify an earlier 
parameter before applying the pattern or the way it's applied depends on 
earlier arguments...

The input is a PATTERN; I want to process it, and apply it, myself.

And it doesn't work anyway; suppose I write:

   >X A *.* C D

How does the program know when the expanded filenames of *.* end, and 
the two extra parameters start? Remember it doesn't know there were four 
parameters, all it seems is one linear stream of arguments. Any 
structure the input may have had is lost.

What happens here:

  >X *.a *.b *.c

It'll just get one big long list of files, not three separate sets.

As I said, it's dumb. And expecting users to stick quotes everywhere is 
not a solution, because that's not going to happen.

> Windows treats the command line as an afterthought, and until a few years
> ago you were limited to a DOS shell. Today, your options are not as
> limited: there's Powershell, and bash for Windows.

At least Windows does it properly. It doesn't even chop the command line 
into different parameters, making it considerably more flexible. (Unless 
you have a program based on a C-style main(nargs,args) entry point where 
the C runtime will do this for you.)

-- 
Bartc


From rosuav at gmail.com  Mon Dec  5 07:23:22 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 5 Dec 2016 23:23:22 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o23jo4$30k$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
Message-ID: <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>

On Mon, Dec 5, 2016 at 10:42 PM, BartC <bc at freeuk.com> wrote:
> At least Windows does it properly. It doesn't even chop the command line
> into different parameters, making it considerably more flexible. (Unless you
> have a program based on a C-style main(nargs,args) entry point where the C
> runtime will do this for you.)

Yes, because there's no way that you can ever get security problems
from improperly parsing command-line arguments. That's why the
recommended way to create a subprocess is os.system(), not the Popen
calls that take a list of already-separated parameters. Right?

ChrisA


From bc at freeuk.com  Mon Dec  5 09:11:10 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 14:11:10 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.30.1480940611.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
Message-ID: <o23sfi$1bn$1@dont-email.me>

On 05/12/2016 12:23, Chris Angelico wrote:
> On Mon, Dec 5, 2016 at 10:42 PM, BartC <bc at freeuk.com> wrote:
>> At least Windows does it properly. It doesn't even chop the command line
>> into different parameters, making it considerably more flexible. (Unless you
>> have a program based on a C-style main(nargs,args) entry point where the C
>> runtime will do this for you.)
>
> Yes, because there's no way that you can ever get security problems
> from improperly parsing command-line arguments.

And you will never get any problems if a program expects 3 parameters 
but instead gets some arbitrary number of arguments, perhaps thousands, 
if one happens to be *, including some that could coincide with some 
actual meaningful input that the program recognises.

  That's why the
> recommended way to create a subprocess is os.system(), not the Popen
> calls that take a list of already-separated parameters. Right?

And nothing will ever go wrong with incorrectly calling Popen that 
takes, if I counted them correctly, up to 14 different parameters?

BTW what does Popen() do when one argument is '*.*'? Will that get 
expanded to multiple extra arguments, and at what point will it be 
expanded?

(I tried to test it, but:

   import subprocess
   subprocess.Popen("python")

didn't work under Linux: 'No such file or directory'. It works under 
Windows but I wanted to see what it did with a parameter *.

Another difference.)

-- 
Bartc


From clvanwall at gmail.com  Mon Dec  5 09:45:53 2016
From: clvanwall at gmail.com (clvanwall)
Date: Mon, 05 Dec 2016 08:45:53 -0600
Subject: When will they fix Python _dbm?
Message-ID: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>

I have been a Perl programmer for 15+ years and decided to give Python a try. ?My platform is windows and I installed the latest 3.5.2. Next I decided to convert a perl program that uses a ndbm database since according to the doc on python, it should be able to work with it. ?Needless to say, I get:?dbm.error: db type is dbm.ndbm, but the module is not available
Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012! ?That's a long time for a bug to be left open. ?
John Van Walleghen


Sent from my Galaxy Tab? A

From rosuav at gmail.com  Mon Dec  5 10:05:50 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 02:05:50 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o23sfi$1bn$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
Message-ID: <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>

On Tue, Dec 6, 2016 at 1:11 AM, BartC <bc at freeuk.com> wrote:
>
> BTW what does Popen() do when one argument is '*.*'? Will that get expanded
> to multiple extra arguments, and at what point will it be expanded?

Nope. Popen is not a shell.

It sounds as if you want a nerfed shell. Go ahead! I'm sure one
exists. It'll frustrate you no end once you get used to a better
shell, though - always does when I find myself on Windows...

ChrisA


From p.f.moore at gmail.com  Mon Dec  5 10:11:18 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 5 Dec 2016 07:11:18 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o23sfi$1bn$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
Message-ID: <33609a46-d6d0-4d1d-a9a9-47ec0689f179@googlegroups.com>

On Monday, 5 December 2016 14:11:34 UTC, BartC  wrote:
> On 05/12/2016 12:23, Chris Angelico wrote:
> > On Mon, Dec 5, 2016 at 10:42 PM, BartC <bc at freeuk.com> wrote:
> >> At least Windows does it properly. It doesn't even chop the command line
> >> into different parameters, making it considerably more flexible. (Unless you
> >> have a program based on a C-style main(nargs,args) entry point where the C
> >> runtime will do this for you.)
> >
> > Yes, because there's no way that you can ever get security problems
> > from improperly parsing command-line arguments.
> 
> And you will never get any problems if a program expects 3 parameters 
> but instead gets some arbitrary number of arguments, perhaps thousands, 
> if one happens to be *, including some that could coincide with some 
> actual meaningful input that the program recognises.

Windows and Linux are different. Neither is unambiguously "right" nor is either unambiguously "wrong". In both cases you need to understand what happens when you type a command, or you *will* get caught out by corner cases.

Calling either approach "dumb" is neither helpful nor productive.

For this specific example, of a program that takes a fragment of SQL as its command line, is one that's very hard to handle cleanly in a cross-platform way, because you actually don't want the shell, or the application, to interpret your arguments for you. The "best" approach is often to accept the SQL command as a single argument (argv[1]) and rely on your users quoting the argument appropriately. Admittedly, that simply pushes the problem onto your users, who may well also be uncomfortable with the subtleties of command line parsing, but at least they are using their choice of shell, so they have a chance of knowing.

The alternative, if you *really* don't want to force your users to understand shell parsing, is to prompt the user for the SQL - either as a simple console input, or (for users who are really uncomfortable with the command line) via a GUI program and a dialog box.

But criticising the (entirely valid, simply different) choices of another OS vendor as "dumb" isn't acceptable, nor is it a way to get to a solution to your issue.

Paul


From p.f.moore at gmail.com  Mon Dec  5 10:17:49 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 5 Dec 2016 07:17:49 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.34.1480950353.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
Message-ID: <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>

On Monday, 5 December 2016 15:06:05 UTC, Chris Angelico  wrote:
> On Tue, Dec 6, 2016 at 1:11 AM, BartC <bc at freeuk.com> wrote:
> >
> > BTW what does Popen() do when one argument is '*.*'? Will that get expanded
> > to multiple extra arguments, and at what point will it be expanded?
> 
> Nope. Popen is not a shell.
> 
> It sounds as if you want a nerfed shell. Go ahead! I'm sure one
> exists. It'll frustrate you no end once you get used to a better
> shell, though - always does when I find myself on Windows...
> 
> ChrisA

For a non-nerfed (but *radically* different to bash) Windows shell, try Powershell. You'll probably hate it, but not because it's limited in capabilities :-)

Paul

PS Apparently, powershell is now open source and available for Linux and OSX. See https://github.com/PowerShell/PowerShell - although I don't know if all the features available on Windows exist on other platforms (yet).


From rosuav at gmail.com  Mon Dec  5 10:30:51 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 02:30:51 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
Message-ID: <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>

On Tue, Dec 6, 2016 at 2:17 AM, Paul  Moore <p.f.moore at gmail.com> wrote:
> For a non-nerfed (but *radically* different to bash) Windows shell, try Powershell. You'll probably hate it, but not because it's limited in capabilities :-)
>

Radically different from every shell I've ever called a shell. It
looks and feels more like a scripting language than a shell. Here's a
Windows batch script I put together that uses PS to do some of its
work:

https://github.com/Rosuav/Gypsum/blob/master/get_gypsum.bat

Granted, that was put together simply by searching Stack Overflow for
"how do I do *this* in PowerShell", so there might be other ways to do
it; but it feels to me like a VBScript interpreter, not a shell.

But hey. It exists by default on recent-enough Windowses, has the
features I want, and can be invoked by someone double-clicking on a
batch file. If you call it "flurgle", it won't make any difference to
me or to my work.

ChrisA


From bc at freeuk.com  Mon Dec  5 10:41:45 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 15:41:45 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.34.1480950353.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
Message-ID: <o241pc$l69$1@dont-email.me>

On 05/12/2016 15:05, Chris Angelico wrote:
> On Tue, Dec 6, 2016 at 1:11 AM, BartC <bc at freeuk.com> wrote:
>>
>> BTW what does Popen() do when one argument is '*.*'? Will that get expanded
>> to multiple extra arguments, and at what point will it be expanded?
>
> Nope. Popen is not a shell.
>
> It sounds as if you want a nerfed shell. Go ahead! I'm sure one
> exists. It'll frustrate you no end once you get used to a better
> shell, though - always does when I find myself on Windows...

That's not the point I was making.

Say you have this program a.py:

   import sys
   print (sys.argv)

And let's say there are just 3 files in the current directory: a.py, 
b.py and c.py.

If run from a Linux shell:

   python a.py *

The output is: ['a.py', 'b.py', 'c.py'] or something along those lines 
(there might be two copies of a.py).

Are you saying that if someone executes:

   subprocess.Popen(["python","a.py", "*"])

the output will be: ['a.py','*']?

In that case forget Windows vs. Linux, you now have a program that will 
get command parameters processed differently depending on whether it was 
invoked from a shell or not.

Or a program that sometimes will see "*" as an argument, and sometimes a 
big list of files that merges into all the other arguments.

-- 
bartc


From amamaenko at gmail.com  Mon Dec  5 10:52:14 2016
From: amamaenko at gmail.com (Anton Mamaenko)
Date: Mon, 5 Dec 2016 18:52:14 +0300
Subject: When will they fix Python _dbm?
In-Reply-To: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
References: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
Message-ID: <A2550B0F-DACF-4445-ACAB-74AC4FC14912@gmail.com>

What is your operating system, environment, and Python build? 

dbm is just a module that might not have been included into your Python build. It's not a bug but a deliberate choice of the package maker. 

Regards,
       Anton

> On 5 Dec 2016, at 17:45, clvanwall <clvanwall at gmail.com> wrote:
> 
> I have been a Perl programmer for 15+ years and decided to give Python a try.  My platform is windows and I installed the latest 3.5.2. Next I decided to convert a perl program that uses a ndbm database since according to the doc on python, it should be able to work with it.  Needless to say, I get: dbm.error: db type is dbm.ndbm, but the module is not available
> Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!  That's a long time for a bug to be left open.  
> John Van Walleghen
> 
> 
> Sent from my Galaxy Tab? A
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From rosuav at gmail.com  Mon Dec  5 10:53:53 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 02:53:53 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o241pc$l69$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
Message-ID: <CAPTjJmqP9UjwG9pYmr2+RuWWSBZN5DR5hwVPRD=zVAZODJ9wUw@mail.gmail.com>

On Tue, Dec 6, 2016 at 2:41 AM, BartC <bc at freeuk.com> wrote:
>
> Are you saying that if someone executes:
>
>   subprocess.Popen(["python","a.py", "*"])
>
> the output will be: ['a.py','*']?
>
> In that case forget Windows vs. Linux, you now have a program that will get
> command parameters processed differently depending on whether it was invoked
> from a shell or not.

Yes, that is correct. *SHELLS DO STUFF*. If you can't comprehend this,
you should get to know your shell. Try this:

python a.py %PATH%

subprocess.Popen(["python", "a.py", "%PATH%"])

Would you expect those to do the same? If you do, prepare for Windows
to surprise you. If you don't, why do you keep expecting other shells
to do nothing?

ChrisA


From p.f.moore at gmail.com  Mon Dec  5 11:03:58 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 5 Dec 2016 08:03:58 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o241pc$l69$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
Message-ID: <dc86dc36-8b51-4946-bcf9-201c488a0f58@googlegroups.com>

On Monday, 5 December 2016 15:41:59 UTC, BartC  wrote:
> On 05/12/2016 15:05, Chris Angelico wrote:
> > On Tue, Dec 6, 2016 at 1:11 AM, BartC <bc at freeuk.com> wrote:
> >>
> >> BTW what does Popen() do when one argument is '*.*'? Will that get expanded
> >> to multiple extra arguments, and at what point will it be expanded?
> >
> > Nope. Popen is not a shell.
> >
> > It sounds as if you want a nerfed shell. Go ahead! I'm sure one
> > exists. It'll frustrate you no end once you get used to a better
> > shell, though - always does when I find myself on Windows...
> 
> That's not the point I was making.
> 
> Say you have this program a.py:
> 
>    import sys
>    print (sys.argv)
> 
> And let's say there are just 3 files in the current directory: a.py, 
> b.py and c.py.
> 
> If run from a Linux shell:
> 
>    python a.py *
> 
> The output is: ['a.py', 'b.py', 'c.py'] or something along those lines 
> (there might be two copies of a.py).
> 
> Are you saying that if someone executes:
> 
>    subprocess.Popen(["python","a.py", "*"])
> 
> the output will be: ['a.py','*']?
> 
> In that case forget Windows vs. Linux, you now have a program that will 
> get command parameters processed differently depending on whether it was 
> invoked from a shell or not.
> 
> Or a program that sometimes will see "*" as an argument, and sometimes a 
> big list of files that merges into all the other arguments.


Python programs, when started, get a list of arguments via sys.argv.

1. On Linux, the OS primitive for executing a program takes a list of arguments and passes them directly. The user's shell is responsible for splitting a command line into "arguments".
2. On Windows, the OS primitive takes a command line. The application is responsible for splitting it into arguments, if it wants to. Most do, for compatibility with the normal argv convention inherited via C from Unix. Many programs let the C runtime do that splitting for them - I don't recall if Python does, or if it implements its own splitting these days.
3. The Popen class directly passes the supplied argv to the called program. (Technically, it has to do some nasty internal juggling to preserve the argv, but you don't need to care about that).

The program always gets a list of arguments. What *provides* that list to it (the Unix shell, the C/Python runtime, or the caller of Popen) varies. And you need (in more complex cases) to understand how the calling environment constructs that list of arguments if you want to reason about behaviour.

Paul


From bc at freeuk.com  Mon Dec  5 11:23:39 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 16:23:39 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.37.1480953242.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CAPTjJmqP9UjwG9pYmr2+RuWWSBZN5DR5hwVPRD=zVAZODJ9wUw@mail.gmail.com>
 <mailman.37.1480953242.2253.python-list@python.org>
Message-ID: <o2447u$uvf$1@dont-email.me>

On 05/12/2016 15:53, Chris Angelico wrote:
> On Tue, Dec 6, 2016 at 2:41 AM, BartC <bc at freeuk.com> wrote:
>>
>> Are you saying that if someone executes:
>>
>>   subprocess.Popen(["python","a.py", "*"])
>>
>> the output will be: ['a.py','*']?
>>
>> In that case forget Windows vs. Linux, you now have a program that will get
>> command parameters processed differently depending on whether it was invoked
>> from a shell or not.
>
> Yes, that is correct. *SHELLS DO STUFF*. If you can't comprehend this,
> you should get to know your shell. Try this:
>
> python a.py %PATH%
>
> subprocess.Popen(["python", "a.py", "%PATH%"])
>
> Would you expect those to do the same? If you do, prepare for Windows
> to surprise you. If you don't, why do you keep expecting other shells
> to do nothing?

You still don't get point. I write a program P, a native executable. It 
takes command line parameters but exactly what it gets depends on 
whether it's started from a 'shell' or from inside another program? I 
don't want to worry about that stuff or exactly how it is invoked!

 > subprocess.Popen(["python", "a.py", "%PATH%"])

Yes, %...% is an escape sequence. Those % signs are supposed to stand 
out and have been chosen not to clash with typical input.

And the end result of the transformation is, here, also a SINGLE thing; 
any parameter following will still be the second parameter, not the 14771th!

Are you saying that the * in ABC*EF also makes the whole thing some 
escape pattern? And one that could massively expand the number of 
parameters, pushing all the following ones out of the way, and making it 
impossible to discover where these expanded parameters end and the 
normal ones recommence.

If someone had thought this up now, it would rapidly be dismissed as 
being unworkable. But because it's been in Unix/Linux/whatever so long, 
no one can see anything wrong with it!

-- 
Bartc


From rosuav at gmail.com  Mon Dec  5 11:39:43 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 03:39:43 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2447u$uvf$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CAPTjJmqP9UjwG9pYmr2+RuWWSBZN5DR5hwVPRD=zVAZODJ9wUw@mail.gmail.com>
 <mailman.37.1480953242.2253.python-list@python.org>
 <o2447u$uvf$1@dont-email.me>
Message-ID: <CAPTjJmrvJFbNHtEH_mZZEHqyD-4hbqFTwywsf6yAav-H2KV1pw@mail.gmail.com>

On Tue, Dec 6, 2016 at 3:23 AM, BartC <bc at freeuk.com> wrote:
> You still don't get point. I write a program P, a native executable. It
> takes command line parameters but exactly what it gets depends on whether
> it's started from a 'shell' or from inside another program? I don't want to
> worry about that stuff or exactly how it is invoked!
>
>> subprocess.Popen(["python", "a.py", "%PATH%"])
>
> Yes, %...% is an escape sequence. Those % signs are supposed to stand out
> and have been chosen not to clash with typical input.
>
> And the end result of the transformation is, here, also a SINGLE thing; any
> parameter following will still be the second parameter, not the 14771th!
>
> Are you saying that the * in ABC*EF also makes the whole thing some escape
> pattern? And one that could massively expand the number of parameters,
> pushing all the following ones out of the way, and making it impossible to
> discover where these expanded parameters end and the normal ones recommence.

Yes. That is exactly what I am saying. Also, try this:

set ENV_VAR=test1 test2
python a.py arg1 %ENV_VAR% arg2

How many args do you get? Is the end result really a single thing? Is
that single-thing-ness consistent across applications?

Windows's cmd.exe is not as simple as you think it is. Linux's bash is
not as insane as you think it is. Both of them take user input,
*MANIPULATE IT*, and pass it along. Keep on making these assumptions
and we will keep on proving them false.

ChrisA


From lew.pitcher at digitalfreehold.ca  Mon Dec  5 11:41:22 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Mon, 05 Dec 2016 11:41:22 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
Message-ID: <o24599$3f0$1@dont-email.me>

On Monday December 5 2016 10:41, in comp.lang.python, "BartC" <bc at freeuk.com>
wrote:

> On 05/12/2016 15:05, Chris Angelico wrote:
>> On Tue, Dec 6, 2016 at 1:11 AM, BartC <bc at freeuk.com> wrote:
>>>
>>> BTW what does Popen() do when one argument is '*.*'? Will that get
>>> expanded to multiple extra arguments, and at what point will it be
>>> expanded?
>>
>> Nope. Popen is not a shell.
>>
>> It sounds as if you want a nerfed shell. Go ahead! I'm sure one
>> exists. It'll frustrate you no end once you get used to a better
>> shell, though - always does when I find myself on Windows...
> 
> That's not the point I was making.
> 
> Say you have this program a.py:
> 
>    import sys
>    print (sys.argv)
> 
> And let's say there are just 3 files in the current directory: a.py,
> b.py and c.py.
> 
> If run from a Linux shell:
> 
>    python a.py *
> 
> The output is: ['a.py', 'b.py', 'c.py'] or something along those lines
> (there might be two copies of a.py).

And, that's because, before invoking Popen, the SHELL has globbed that '*'
argument into the three filenames and substituted those names where the '*'
was.

If you don't use a shell, then (on Unix), you have to perform the globbing
yourself before invoking Popen.

> 
> Are you saying that if someone executes:
> 
>    subprocess.Popen(["python","a.py", "*"])
> 
> the output will be: ['a.py','*']?
> 
> In that case forget Windows vs. Linux, you now have a program that will
> get command parameters processed differently depending on whether it was
> invoked from a shell or not.

Yes.

> Or a program that sometimes will see "*" as an argument, and sometimes a
> big list of files that merges into all the other arguments.
> 

Yes.

-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From steve+python at pearwood.info  Mon Dec  5 11:49:04 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 06 Dec 2016 03:49:04 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
Message-ID: <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Mon, 5 Dec 2016 10:42 pm, BartC wrote:

> I don't know what a shell is. To me, it's some sort of user interface to
> the OS. 

https://en.wikipedia.org/wiki/Unix_shell

You've never used cmd.com or command.exe? "The DOS prompt"? That's
(effectively) a shell.

Pedants may wish to explain exactly why the DOS prompt isn't a shell but to
a first approximation I think its close enough.

And yes, that's exactly what it is: its a text-based user interface to the
OS. And like any user-interface, designers can choose different user
interfaces, which will vary in power and convenience. And in the Unix/Linux
world, the command shell is not just a text interface, its a powerful
command interpreter and programming language.



> So if someone types: 
> 
>    > X A B C
> 
> You would expect X to be launched, and be given arguments A, B and C.

Would I? I don't think so.

Even the DOS prompt supports some level of globbing. Its been a while since
I've used the DOS prompt in anger, but I seem to recall being able to do
things like:

dir a*

to get a listing of all the files starting with "a". So *something* is
treating the * as a special character. In Linux and Unix, that's invariably
the shell, before the dir command even sees what you typed.

In DOS, it might be the dir command itself. The disadvantage of the DOS way
of doing this is that *every single command and application* has to
re-implement its own globbing, very possibly inconsistently. That's a lot
of duplicated work and re-inventing the wheel, and the user will never know
what 

    some_program a*

will do. Will it operate on all the files in the current directory that
start with "a"? Or will it try to operate on a single file called
literally "a*"? Which of course won't exist because * is a reserved
character on DOS/Windows file systems. You can't know ahead of time unless
you study the manual to see what metacharacters this specific command
supports.

The Unix way is far more consistent: applications typically don't have to
care about globbing, because the shell handles glob expansion, environment
variables, etc.

[Aside: if you take the big picture, the Unix/Linux way is probably LESS
consistent, because you can have any number of shells (sh, ash, bash, csh,
tcsh, dash, hush, zsh, and I expect many more). But in practice, there's
one lowest-common-denominator standard (sh) and one major de facto standard
(bash), and most of the shells are supersets of the original sh, so simple
things like wildcards behave in pretty similar ways.]

The downside of this is that if you don't want metacharacters expanded, you
have to tell the shell to ignore it. The easiest way is to escape them with
a backslash, or quote the string. But of course this being Unix, its
completely configurable (using an obscure syntax that's different for every
shell):

http://stackoverflow.com/questions/11456403/stop-shell-wildcard-character-expansion



> You wouldn't expect any of them to be expanded to some unknown number of
> arguments.

Actually yes I would. If they could be interpreted as file names with
globbing or environment variables, that's exactly what I would expect.

Even at the DOS prompt.

And I'd be completely gobsmacked if (say) the dir command understood the ?
metacharacter but the cp command didn't.


> In the same way that in code, you don't expect X(A,B,C) to be expanded
> to X(A,B0,B1,B2,B3,B4,B5,....., C) if B happens to specify a slice.

In Python? No, I wouldn't expect that. Python's not a shell, and the design
is different. In Python, you have to use the metacharacter * to expand a
single argument into multiple arguments.



>> Did you think it was a regular character like 'a' and 'z'?
> 
> If one of the parameters was a regular expression, would you expect it
> to be expanded to the entire possible set of inputs that match the
> expression?

No, because Unix shells use globs, not regexes. Just like the DOS prompt.
Globs are simpler and require less typing, something system administrators
appreciate because (unlike programmers) interactive commands are written
far more than they are read, so brevity is appreciated.

(That doesn't excuse the umount command though. Really, you couldn't include
the "n"?)

So yes, I would expect that if I said

dir a*

I would get a listing of all the files starting with "a", not just the
single file called literally "a*".



>> I think it boils down to what the user expects. Linux and Unix users tend
>> to be technically-minded folks who use the command line a lot and demand
>> powerful tools, and they expect that wildcards like * should be expanded.
> 
> Which is dumb. How does the shell know exactly what I want to do with
> *.* or f*.c or whatever? Perhaps I don't want the expanded list right 
> now (all the 3.4 million elements); 

Sure, no problem. Just escape the metacharacters so they aren't expanded,
just as in Python if you want the literal string backslash n you can
write "\\n" to escape the backslash metacharacter.


> perhaps I want to apply the same pattern to several directories; 

  ls {foo,bar,baz/foobar}/*.*


is equivalent to:

  ls foo/*.* bar/*.* baz/foobar/*.*


And if I want a file that is *literally* called star dot star from any of
those directories:

  ls {foo,bar,baz/foobar}/"*.*"



> perhaps I'm passing it on to another 
> program; perhaps I'm going to be writing it as another script; perhaps I
> just want to print out the parameter list; perhaps I want to transform
> or validate the pattern first; maybe I need to verify an earlier
> parameter before applying the pattern or the way it's applied depends on
> earlier arguments...

Fine. Just escape the damn thing and do whatever you like to it.



> The input is a PATTERN; I want to process it, and apply it, myself.

When you double-click on a .doc file, and Windows launches Word and opens
the file for editing, do you rant and yell that you didn't want Word to
open the file, you wanted to process the file name yourself?

Probably not. You probably consider it perfectly normal and unobjectionable
that when you double click a file, Windows:

- locates the file association for that file;
- launches that application (if its not already running);
- instructs the application to open the file;

(and whatever else needs to be done). That's what double clicking is *for*,
and if you don't like it, you shouldn't be using Windows.

Well, in the Unix world, the shells are designed for the benefit of system
administrators who are *mostly* dealing with file names.  Regardless of
what you think of it, they *want* this behaviour. For forty plus years,
Unix system administrators have been extremely productive using this model,
and Microsoft has even brought the Linux bash shell to Windows.

The bottom line is this:

In your application, if you receive an argument *.*, what do you do with it?
You probably expand it yourself. How does the application know when not to
expand the wild cards? You need to support some sort of command-line switch
to turn it off, but that will effect the entire command line. So you need
some sort of escaping mechanism so that you can pass

myprogram *.* *.*

and have the first *.* expanded but not the second. (For example.)

Congratulations, you've just re-invented your own mini-shell.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From marko at pacujo.net  Mon Dec  5 11:53:50 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Mon, 05 Dec 2016 18:53:50 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
 <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>
 <mailman.35.1480951854.2253.python-list@python.org>
Message-ID: <877f7es4f5.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Tue, Dec 6, 2016 at 2:17 AM, Paul  Moore <p.f.moore at gmail.com> wrote:
>> For a non-nerfed (but *radically* different to bash) Windows shell,
>> try Powershell. You'll probably hate it, but not because it's limited
>> in capabilities :-)
>
> Radically different from every shell I've ever called a shell. It
> looks and feels more like a scripting language than a shell.

In recent years, I've been disillusioned with bash and started using
Python more and more where I would previously have used bash. Python's
explicit syntax does automatically give you a level of security, but I
must say the subprocess.Popen.communicate syntax is painful as hell.
Your traditional one-liners turn into five-liners, and a casual observer
will have a slightly hard time understanding what's going on.


Marko


From rosuav at gmail.com  Mon Dec  5 12:06:09 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 04:06:09 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <877f7es4f5.fsf@elektro.pacujo.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
 <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>
 <mailman.35.1480951854.2253.python-list@python.org>
 <877f7es4f5.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmpMZuyc4m7kt6t+0AQuLRnduSaUzSHAgFxYBr1OL_AWGw@mail.gmail.com>

On Tue, Dec 6, 2016 at 3:53 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> In recent years, I've been disillusioned with bash and started using
> Python more and more where I would previously have used bash. Python's
> explicit syntax does automatically give you a level of security, but I
> must say the subprocess.Popen.communicate syntax is painful as hell.
> Your traditional one-liners turn into five-liners, and a casual observer
> will have a slightly hard time understanding what's going on.

Congratulations. You've just discovered why bash is useful.

ChrisA


From steve+python at pearwood.info  Mon Dec  5 12:12:01 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 06 Dec 2016 04:12:01 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
Message-ID: <58459fe3$0$1588$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 02:41 am, BartC wrote:

> In that case forget Windows vs. Linux, you now have a program that will
> get command parameters processed differently depending on whether it was
> invoked from a shell or not.

Er, yeah? You say that as if it were a bad thing.

Look at it this way. Suppose you were a really cunning system administrator
who spent 8 or 10 hours a day typing in commands and arguments that were
frequently file names. So you built yourself a macro system where you type
a command:

    foo bar baz

and the macro would walk through the command line, automatically expanding
environment variables and file name globs. And because you used this 95% of
the time, and it was intended as a convenience for interactive use, you
linked the macro to the Enter key, so that only a single key press was
needed to do this processing.

But of course you're not an idiot, you know full well that there are
occasions where you want to avoid the expansion. So you include commands to
disable and enable that macro, and to give you even more fine-grained
control of what is expanded where, you build it escaping mechanisms so that
you can expand part of the command line and not other parts.

And then I come along, full of righteous indignation, and start yelling

"Wait a minute, now your application gets its command parameters processed
differently depend on whether you called it from C or used your macro!"

And you would answer:

"Of course it does, that's the whole bloody point of the macro!"

(By the way, I think that you will find that when you call Popen, if you set
shell=True it will be invoked in a subshell, which means you'll get the
full shell experience including command expansion. For good and evil.
That's a security risk, if you're getting the arguments from an untrusted
source, so don't pass shell=True unless you know what you're doing.)


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From walters.justin01 at gmail.com  Mon Dec  5 12:13:59 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Mon, 5 Dec 2016 09:13:59 -0800
Subject: When will they fix Python _dbm?
In-Reply-To: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
References: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
Message-ID: <CAO1D73Hx67CcxspBd_jmaXxN24oaB6Dhyvc0uJysoVN0Xop-5w@mail.gmail.com>

On Mon, Dec 5, 2016 at 6:45 AM, clvanwall <clvanwall at gmail.com> wrote:

> I have been a Perl programmer for 15+ years and decided to give Python a
> try.  My platform is windows and I installed the latest 3.5.2. Next I
> decided to convert a perl program that uses a ndbm database since according
> to the doc on python, it should be able to work with it.  Needless to say,
> I get: dbm.error: db type is dbm.ndbm, but the module is not available
> Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!
> That's a long time for a bug to be left open.
> John Van Walleghen
>
>
> Sent from my Galaxy Tab? A
> --
> https://mail.python.org/mailman/listinfo/python-list
>


Hi there,

Could you please post code that is giving you an error?


From skip.montanaro at gmail.com  Mon Dec  5 12:14:12 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 5 Dec 2016 11:14:12 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CANc-5UwWJ7T7k43=cgPsyQuQh7qn4LbCYa=8Hqr2BVdRgd4PMg@mail.gmail.com>

On Mon, Dec 5, 2016 at 10:49 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
>
> In DOS, it might be the dir command itself. The disadvantage of the DOS way
> of doing this is that *every single command and application* has to
> re-implement its own globbing, very possibly inconsistently. That's a lot
> of duplicated work and re-inventing the wheel, and the user will never know
> what
>
>     some_program a*
>
> will do.

ISTR that the way DOS/Windows operate at the text prompt level was
modeled on VMS. As you indicated, each command was responsible for its
own "globbing". I've never programmed in DOS or Windows, and its been
decades since I programmed in VMS, but I imagine that both
environments probably provide some standard sort of globbing library.

On an only peripherally related note, I was reminded this morning of
how some/many GUI environments try to protect people from themselves.
I am just now coming up to speed in a new job which provides me with a
Windows window onto an otherwise Linux development environment. I
tried starting the X server this morning (something called Xming), and
it complained about not being able to write its log file (I suspect
Xming was alread. I tried to navigate to that location through the
Computer doohickey (Explorer?) but couldn't get there. The program (or
more likely the program's programmers) had decided that I had no
business "exploring" into my AppData folder. To get there, I had to
drop into a command prompt.

So, another vote for a text/shell interface. It gives you enough rope
to hang yourself, but assumes you won't, because, "we're all adults
here." I do understand why Windows hides stuff from users in the GUI
though. As a webmaster at python.org monitor, I can attest to the
relatively large number of questions received there asking about
removing Python "because I don't use it for anything." :-) This
started happening about the time the long defunct Compaq started to
write admin tools for Windows in Python.

Skip


From marko at pacujo.net  Mon Dec  5 12:23:31 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Mon, 05 Dec 2016 19:23:31 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
 <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>
 <mailman.35.1480951854.2253.python-list@python.org>
 <877f7es4f5.fsf@elektro.pacujo.net>
 <CAPTjJmpMZuyc4m7kt6t+0AQuLRnduSaUzSHAgFxYBr1OL_AWGw@mail.gmail.com>
 <mailman.39.1480957578.2253.python-list@python.org>
Message-ID: <87y3zuqoh8.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Tue, Dec 6, 2016 at 3:53 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> In recent years, I've been disillusioned with bash and started using
>> Python more and more where I would previously have used bash.
>> Python's explicit syntax does automatically give you a level of
>> security, but I must say the subprocess.Popen.communicate syntax is
>> painful as hell. Your traditional one-liners turn into five-liners,
>> and a casual observer will have a slightly hard time understanding
>> what's going on.
>
> Congratulations. You've just discovered why bash is useful.

Bash is nice, too nice. It makes it easy to write code that's riddled
with security holes. The glorious Unix tradition is to ignore the
pitfalls and forge ahead come what may.


Marko


From rosuav at gmail.com  Mon Dec  5 12:35:09 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 04:35:09 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <87y3zuqoh8.fsf@elektro.pacujo.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
 <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>
 <mailman.35.1480951854.2253.python-list@python.org>
 <877f7es4f5.fsf@elektro.pacujo.net>
 <CAPTjJmpMZuyc4m7kt6t+0AQuLRnduSaUzSHAgFxYBr1OL_AWGw@mail.gmail.com>
 <mailman.39.1480957578.2253.python-list@python.org>
 <87y3zuqoh8.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmrb72_bFzCK_PdmV+8GzzAFWi25afmYerCouwrUkmo6rw@mail.gmail.com>

On Tue, Dec 6, 2016 at 4:23 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Chris Angelico <rosuav at gmail.com>:
>
>> On Tue, Dec 6, 2016 at 3:53 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>>> In recent years, I've been disillusioned with bash and started using
>>> Python more and more where I would previously have used bash.
>>> Python's explicit syntax does automatically give you a level of
>>> security, but I must say the subprocess.Popen.communicate syntax is
>>> painful as hell. Your traditional one-liners turn into five-liners,
>>> and a casual observer will have a slightly hard time understanding
>>> what's going on.
>>
>> Congratulations. You've just discovered why bash is useful.
>
> Bash is nice, too nice. It makes it easy to write code that's riddled
> with security holes. The glorious Unix tradition is to ignore the
> pitfalls and forge ahead come what may.

Bash assumes that the person typing commands has the full power to
execute commands. I'm not sure what you mean by "security holes",
unless it's passing text through bash that came from people who aren't
allowed to type commands.

ChrisA


From bc at freeuk.com  Mon Dec  5 13:02:12 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 18:02:12 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o24a0n$m62$1@dont-email.me>

On 05/12/2016 16:49, Steve D'Aprano wrote:
> On Mon, 5 Dec 2016 10:42 pm, BartC wrote:

>> So if someone types:
>>
>>    > X A B C
>>
>> You would expect X to be launched, and be given arguments A, B and C.
>
> Would I? I don't think so.
>
> Even the DOS prompt supports some level of globbing. Its been a while since
> I've used the DOS prompt in anger, but I seem to recall being able to do
> things like:
>
> dir a*
>
> to get a listing of all the files starting with "a". So *something* is
> treating the * as a special character. In Linux and Unix, that's invariably
> the shell, before the dir command even sees what you typed.
>
> In DOS, it might be the dir command itself.

Yes, it has to be, as there is probably no space to first construct an 
in-memory list of all the files.

  The disadvantage of the DOS way
> of doing this is that *every single command and application* has to
> re-implement its own globbing, very possibly inconsistently. That's a lot
> of duplicated work and re-inventing the wheel,

Which will need to be done anyway. Expansion of filespecs with wildcards 
may need to be done from inside a program.

On Windows that involves calling FindFirstFile/FindNextFile (which takes 
care of wildcards for you), and on Linux it might be opendir/readdir 
(which doesn't; you have to call fnmatch to accept/reject each file).

(I had to port such code recently across OSes for my language; on both 
systems, dirlist(filespec) returns a list of files matching the wildcard 
specification provided. No shell expansion is needed!)

> The Unix way is far more consistent: applications typically don't have to
> care about globbing, because the shell handles glob expansion, environment
> variables, etc.

It's not consistent because program P will sometimes see * and sometimes 
a list of files. On Windows P will /never/ see a list of files if the 
start point is *. Not without a lot of intervention.

And I've already posted a long list of reasons why Linux shell's 
behaviour is undesirable, as you want to literal argument, or you want 
to do something with the filespec that is different from what the shell 
will do, or you want to do it later (perhaps the files question DON'T 
EXIST until halfway through the program).

OK I'm just thinking up more reasons why I don't like Linux shell's 
behaviour.

>> You wouldn't expect any of them to be expanded to some unknown number of
>> arguments.
>
> Actually yes I would. If they could be interpreted as file names with
> globbing or environment variables, that's exactly what I would expect.

If the syntax is:

   program filespec

or:

   program filespec file

how do you tell whether the last file in an argument list is the 
optional 'file', or the last file of the expansion of 'filespec'?

> So yes, I would expect that if I said
>
> dir a*
>
> I would get a listing of all the files starting with "a", not just the
> single file called literally "a*".

So what does 'dir' do then; just print?

Since it sounds like:

   echo *.*

would do the job just as well! (If 'echo' is a program that just lists 
its input.)

> Fine. Just escape the damn thing and do whatever you like to it.

I'm not the one running the program. Perhaps you don't know how stupid 
users can be.

>> The input is a PATTERN; I want to process it, and apply it, myself.
>
> When you double-click on a .doc file,

Are we still talking about a console or terminal here? Any filenames 
displayed as part of the dialogue (result of ls or dir for example) are 
usually not clickable.

> and Windows launches Word and opens
> the file for editing, do you rant and yell that you didn't want Word to
> open the file, you wanted to process the file name yourself?

GUIs are different.

-- 
Bartc


From rosuav at gmail.com  Mon Dec  5 13:21:42 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 05:21:42 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24a0n$m62$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o24a0n$m62$1@dont-email.me>
Message-ID: <CAPTjJmqHOW6g3_Jg7KDgGgXKXRwSH7Yhm808dR9dL2yueZrkmw@mail.gmail.com>

On Tue, Dec 6, 2016 at 5:02 AM, BartC <bc at freeuk.com> wrote:
> If the syntax is:
>
>   program filespec
>
> or:
>
>   program filespec file
>
> how do you tell whether the last file in an argument list is the optional
> 'file', or the last file of the expansion of 'filespec'?

Why should you care? I have used shell globbing to pass precisely two
parameters to a program. More often, I use this syntax, which Windows
simply doesn't support:

ffmpeg -i some-really-long-file-name.{avi,mkv}

to convert a file from one format to another. And if the .mkv file
already exists, I can just use ".*" at the end, although that does
depend on them being in alphabetical order. The shell is predictable
and therefore useful. This trick is guaranteed to work, no matter what
program I'm using. On Windows, I have to hope that the program expands
these notations correctly.

ChrisA


From eryksun at gmail.com  Mon Dec  5 13:34:55 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 5 Dec 2016 18:34:55 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o241pc$l69$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
Message-ID: <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>

On Mon, Dec 5, 2016 at 3:41 PM, BartC <bc at freeuk.com> wrote:
>
> Are you saying that if someone executes:
>
>   subprocess.Popen(["python","a.py", "*"])
>
> the output will be: ['a.py','*']?
>
> In that case forget Windows vs. Linux, you now have a program that will get
> command parameters processed differently depending on whether it was invoked
> from a shell or not.
>
> Or a program that sometimes will see "*" as an argument, and sometimes a big
> list of files that merges into all the other arguments.

If it sees "*", it will try to open a file named "*". That's a valid
filename in Unix, but it should be avoided. We wouldn't want someone
to accidentally run `rm *` instead of `rm \*`.

In Windows, it's invalid for filenames to include wildcard characters
(i.e. '*' and '?' as well as the MS-DOS compatibility wildcards '<',
'>', and '"' -- DOS_STAR, DOS_QM, and DOS_DOT). Since there's no
ambiguity of intent, if you've linked an executable with
[w]setargv.obj, the C runtime will happily expand "*" in the argv
list.

I don't understand your concern regarding Unix shell scripting syntax.
On Windows, are you as troubled that environment variables such as
%USERNAME% get expanded by cmd.exe, but not by CreateProcess? Does it
bother you that cmd consumes its "^" escape character if it's not
escaped as "^^"? For example:

    C:\>python.exe -i -c "" remove: ^ keep: ^^
    >>> import win32api
    >>> win32api.GetCommandLine()
    'python.exe  -i -c "" remove:  keep: ^'

Every command-line shell that I've ever used is also a quirky
scripting language. Shell literacy requires learning at least the
syntax and operators of the language.


From gheskett at shentel.net  Mon Dec  5 13:37:39 2016
From: gheskett at shentel.net (Gene Heskett)
Date: Mon, 5 Dec 2016 13:37:39 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <87y3zuqoh8.fsf@elektro.pacujo.net>
References: <o21u2p$co1$1@dont-email.me>
 <mailman.39.1480957578.2253.python-list@python.org>
 <87y3zuqoh8.fsf@elektro.pacujo.net>
Message-ID: <201612051337.39947.gheskett@shentel.net>

On Monday 05 December 2016 12:23:31 Marko Rauhamaa wrote:

> Chris Angelico <rosuav at gmail.com>:
> > On Tue, Dec 6, 2016 at 3:53 AM, Marko Rauhamaa <marko at pacujo.net> 
wrote:
> >> In recent years, I've been disillusioned with bash and started
> >> using Python more and more where I would previously have used bash.
> >> Python's explicit syntax does automatically give you a level of
> >> security, but I must say the subprocess.Popen.communicate syntax is
> >> painful as hell. Your traditional one-liners turn into five-liners,
> >> and a casual observer will have a slightly hard time understanding
> >> what's going on.
> >
> > Congratulations. You've just discovered why bash is useful.
>
> Bash is nice, too nice. It makes it easy to write code that's riddled
> with security holes. The glorious Unix tradition is to ignore the
> pitfalls and forge ahead come what may.
>
>
> Marko

I like bash, use it a lot.  But the ultimate language (damn the security 
considerations real or imagined) is still ARexx. The *nixes have not, 20 
years later, a do it all language that can play on the same field with 
ARexx.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>


From vmahajan at centerpointmedia.com  Mon Dec  5 13:39:46 2016
From: vmahajan at centerpointmedia.com (vmahajan at centerpointmedia.com)
Date: Mon, 5 Dec 2016 10:39:46 -0800 (PST)
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
In-Reply-To: <mailman.171.1479327928.30068.python-list@python.org>
References: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
 <o0ifbb$mc9$1@blaine.gmane.org>
 <mailman.171.1479327928.30068.python-list@python.org>
Message-ID: <fafad656-6c14-4714-b810-4c21674cbcf2@googlegroups.com>

On Wednesday, November 16, 2016 at 3:25:39 PM UTC-5, Peter Otten wrote:
> vmahajan at centerpointmedia.com wrote:
> 
> > I am running Python2.7, wherein I am running the following price of code:
> > 
> > y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
> > print ('Predictions: {}'.format(str(y)))
> > 
> > The output of the following is ""<generator object _as_iterable at
> > 0x7f0476373e10>"
> > 
> > However, the desired output must be in the format [1 0 0 1 1 0 0 1].
> > 
> > Can someone help me print the output in this format
> 
> You give no context, but my suggestion would be to try and omit the 
> 
> as_iterable=True
> 
> part...


Can someone help me print a generator object?



From bc at freeuk.com  Mon Dec  5 13:50:30 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 18:50:30 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.44.1480960040.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
Message-ID: <o24cr9$1ri$1@dont-email.me>

On 05/12/2016 17:46, Dennis Lee Bieber wrote:
> On Mon, 5 Dec 2016 11:42:08 +0000, BartC <bc at freeuk.com> declaimed the
> following:
>
>
>> And it doesn't work anyway; suppose I write:
>>
>>   >X A *.* C D
>>
>> How does the program know when the expanded filenames of *.* end, and
>> the two extra parameters start? Remember it doesn't know there were four
>> parameters, all it seems is one linear stream of arguments. Any
>> structure the input may have had is lost.
>>
> 	And just what ARE A, C, and D?

It doesn't matter, and is not the concern of the shell. It should 
restrict itself to the basic parsing that may be necessary when 
parameters are separated by white-space and commas, if a parameter can 
contain white-space or commas. That usually involves surrounding the 
parameter with quotes.

One would be very annoyed if, reading a CSV file, where each of N values 
on a line correspond to a field of record, if one entry of "?LBC" 
expanded itself to a dozen entries, screwing everything up.

Shell command line processing shouldn't be attempting anything more than 
that.

> 	If they are command options, on a Linux shell options appear as
>
> X -A *.* -C -D
>
> Even Windows command processor separates optional/command stuff via
>
> X /A *.* /C /D
>
> ... or requires the parameter defining the file(s) to process as the last
> arguments

Nonsense. If I pass that to a Python program that prints sys.argv, I get:

['a.py', '/A', '*.*', '/C', '/D']

Presumably a Windows programmer is grown up enough to make their own 
decisions as to what to do with that input.

All that's needed is one little standard library function to process 
sys.argc, with unexpanded parameters, into a list of expanded arguments, 
if that's really what someone wants (and then needing to trawl through 
it all looking for the options).

Then you get the best of both worlds.


> X A C D *.*

So how do I do:

    gcc *.c -lm

The -lm needs to go at the end.

Presumably it now needs to check each parameter seeing if it resembles a 
file more than it does an option? And are options automatically taken 
care of, or is that something that, unlike the easier wildcards, need to 
be processed explicitly?

-- 
Bartc


From eryksun at gmail.com  Mon Dec  5 13:54:30 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 5 Dec 2016 18:54:30 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <dc86dc36-8b51-4946-bcf9-201c488a0f58@googlegroups.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <dc86dc36-8b51-4946-bcf9-201c488a0f58@googlegroups.com>
Message-ID: <CACL+1asAfL=y5vGELLJePh0dNbW-cOomn2Y6mgeH2PYzOd-zfg@mail.gmail.com>

On Mon, Dec 5, 2016 at 4:03 PM, Paul  Moore <p.f.moore at gmail.com> wrote:
> 2. On Windows, the OS primitive takes a command line. The application is
> responsible for splitting it into arguments, if it wants to. Most do, for
> compatibility with the normal argv convention inherited via C from Unix. Many
> programs let the C runtime do that splitting for them - I don't recall if Python
> does, or if it implements its own splitting these days.

Windows Python uses the CRTs parsed argument list. python.exe simply
uses the wmain entry point:

    int
    wmain(int argc, wchar_t **argv)
    {
        return Py_Main(argc, argv);
    }

pythonw.exe uses the wWinMain entry point and the CRT's __argc and
__wargv variables:

    int WINAPI wWinMain(
        HINSTANCE hInstance,      /* handle to current instance */
        HINSTANCE hPrevInstance,  /* handle to previous instance */
        LPWSTR lpCmdLine,         /* pointer to command line */
        int nCmdShow              /* show state of window */
    )
    {
        return Py_Main(__argc, __wargv);
    }

python[w].exe doesn't link in wsetargv.obj, so it doesn't support
wildcard expansion.


From rosuav at gmail.com  Mon Dec  5 14:01:30 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 06:01:30 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24cr9$1ri$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
Message-ID: <CAPTjJmpxTgnmN9Mp261Yzv_1PqJZTP7qOE_Nb4JPnWf_7BcqWw@mail.gmail.com>

On Tue, Dec 6, 2016 at 5:50 AM, BartC <bc at freeuk.com> wrote:
> So how do I do:
>
>    gcc *.c -lm
>
> The -lm needs to go at the end.
>
> Presumably it now needs to check each parameter seeing if it resembles a
> file more than it does an option? And are options automatically taken care
> of, or is that something that, unlike the easier wildcards, need to be
> processed explicitly?

Actually, gcc processes a series of "things", where each thing could
be a file, a library definition, etc. So it recognizes "-lm" as a
library designation regardless of where it exists. It does NOT have to
be at the end specifically - you can have more file names after it.

ChrisA


From lew.pitcher at digitalfreehold.ca  Mon Dec  5 14:03:21 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Mon, 05 Dec 2016 14:03:21 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CAPTjJmqP9UjwG9pYmr2+RuWWSBZN5DR5hwVPRD=zVAZODJ9wUw@mail.gmail.com>
 <mailman.37.1480953242.2253.python-list@python.org>
 <o2447u$uvf$1@dont-email.me>
Message-ID: <o24dja$230$1@dont-email.me>

On Monday December 5 2016 11:23, in comp.lang.python, "BartC" <bc at freeuk.com>
wrote:

> On 05/12/2016 15:53, Chris Angelico wrote:
>> On Tue, Dec 6, 2016 at 2:41 AM, BartC <bc at freeuk.com> wrote:
>>>
>>> Are you saying that if someone executes:
>>>
>>>   subprocess.Popen(["python","a.py", "*"])
>>>
>>> the output will be: ['a.py','*']?
>>>
>>> In that case forget Windows vs. Linux, you now have a program that will
>>> get command parameters processed differently depending on whether it was
>>> invoked from a shell or not.
>>
>> Yes, that is correct. *SHELLS DO STUFF*. If you can't comprehend this,
>> you should get to know your shell. Try this:
>>
>> python a.py %PATH%
>>
>> subprocess.Popen(["python", "a.py", "%PATH%"])
>>
>> Would you expect those to do the same? If you do, prepare for Windows
>> to surprise you. If you don't, why do you keep expecting other shells
>> to do nothing?
> 
> You still don't get point. I write a program P, a native executable. It
> takes command line parameters but exactly what it gets depends on
> whether it's started from a 'shell' or from inside another program?

In Unix, it always has.


> I 
> don't want to worry about that stuff or exactly how it is invoked!

Then, I guess that you have a problem, don't you?

>  > subprocess.Popen(["python", "a.py", "%PATH%"])
> 
> Yes, %...% is an escape sequence. Those % signs are supposed to stand
> out and have been chosen not to clash with typical input.
> 
> And the end result of the transformation is, here, also a SINGLE thing;
> any parameter following will still be the second parameter, not the 14771th!
> 
> Are you saying that the * in ABC*EF also makes the whole thing some
> escape pattern?

If you ask the shell to parse the arguments, then, "YES, the * in ABC*EF makes
the argument a candidate for globbing".

> And one that could massively expand the number of parameters,

Yes

> pushing all the following ones out of the way, and making it 
> impossible to discover where these expanded parameters end and the
> normal ones recommence.

The best way to think about it is that all parameters are parameters, whether
derived from a glob input to a shell, or explicitly specified in the
invocation.

If you have a need for positional parameters, then either specify that your
code only accepts them in specific places, or find a way to disable globbing
(it can be done) and handle the expansion yourself, in your own code.

> If someone had thought this up now, it would rapidly be dismissed as
> being unworkable. But because it's been in Unix/Linux/whatever so long,
> no one can see anything wrong with it!

Primarily because there /IS/ nothing wrong with it.

-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From torriem at gmail.com  Mon Dec  5 14:08:13 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 12:08:13 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPTjJmqHOW6g3_Jg7KDgGgXKXRwSH7Yhm808dR9dL2yueZrkmw@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o24a0n$m62$1@dont-email.me>
 <CAPTjJmqHOW6g3_Jg7KDgGgXKXRwSH7Yhm808dR9dL2yueZrkmw@mail.gmail.com>
Message-ID: <f970e3a2-1e49-a5d4-de90-1be2dc229ad4@gmail.com>

On 12/05/2016 11:21 AM, Chris Angelico wrote:
> On Tue, Dec 6, 2016 at 5:02 AM, BartC <bc at freeuk.com> wrote:
>> If the syntax is:
>>
>>   program filespec
>>
>> or:
>>
>>   program filespec file
>>
>> how do you tell whether the last file in an argument list is the optional
>> 'file', or the last file of the expansion of 'filespec'?
> 
> Why should you care? I have used shell globbing to pass precisely two
> parameters to a program. More often, I use this syntax, which Windows
> simply doesn't support:
> 
> ffmpeg -i some-really-long-file-name.{avi,mkv}
> 
> to convert a file from one format to another. And if the .mkv file
> already exists, I can just use ".*" at the end, although that does
> depend on them being in alphabetical order. The shell is predictable
> and therefore useful. This trick is guaranteed to work, no matter what
> program I'm using. On Windows, I have to hope that the program expands
> these notations correctly.

Agreed. I do this sort of trick all the time, even when I want to pass
just a single file to a program.  I often use expansion for paths as well:

somecommand /path/to/somelongname*withendpart/myepisode*2x03*mp4

I could use tab expansion as I go, but tab expansion might reveal
several options to pick from, requiring a few additional keystrokes to
arrive at the path I want.  Globs save that typing.  And shells are
smart enough to expand the expression even after several wildcards have
been used.  It's a win win.  And the program doesn't have to know
anything about it to work.

Now I usually can use the same expressions in cmd.exe. But I find other
parts of Windows' command-line parsing to really strange, particularly
when it comes to spaces in the filenames.  I'm not sure if this is
cmd.exe's fault or just the win32 api.

In the Unix world, there are times when you don't want shell expansion.
For example, when dealing with ssh or rsync, you most often want the
wildcards and other expression characters to be passed through to the
remote process.  By default zsh makes you escape them all so they won't
attempt expansion locally.  If zsh can't expand an expression locally,
it gives you an error.  This can be controlled with a config option.

Bash on the other hand, passes through any expressions it can't expand
as is.  So under certain circumstances, "python a*.py" would indeed pass
"a*.py" to the program.  But 99% of the time this does not matter.
Whether the glob expands to zero files or the program attempts to open
"a*.py" as a literal file, the effect is usually the same.  Most unix
shells default to this behavior, which is useful when using rsync.

I'd far rather leave it up to the shell to do expansion than to do it
myself in a program. It allows consistency within the shell experience.
If people want the zsh idea of erring out on an expansion, they can have
that.  Or they can use a shell that behaves like bash.  Either way there
is a consistency there that's just not there on Windows.


From irmen.NOSPAM at xs4all.nl  Mon Dec  5 14:21:44 2016
From: irmen.NOSPAM at xs4all.nl (Irmen de Jong)
Date: Mon, 5 Dec 2016 20:21:44 +0100
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
In-Reply-To: <fafad656-6c14-4714-b810-4c21674cbcf2@googlegroups.com>
References: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
 <o0ifbb$mc9$1@blaine.gmane.org>
 <mailman.171.1479327928.30068.python-list@python.org>
 <fafad656-6c14-4714-b810-4c21674cbcf2@googlegroups.com>
Message-ID: <5845be47$0$21479$e4fe514c@news.xs4all.nl>

On 5-12-2016 19:39, vmahajan at centerpointmedia.com wrote:
> On Wednesday, November 16, 2016 at 3:25:39 PM UTC-5, Peter Otten wrote:
>> vmahajan at centerpointmedia.com wrote:
>>
>>> I am running Python2.7, wherein I am running the following price of code:
>>>
>>> y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
>>> print ('Predictions: {}'.format(str(y)))
>>>
>>> The output of the following is ""<generator object _as_iterable at
>>> 0x7f0476373e10>"
>>>
>>> However, the desired output must be in the format [1 0 0 1 1 0 0 1].
>>>
>>> Can someone help me print the output in this format
>>
>> You give no context, but my suggestion would be to try and omit the 
>>
>> as_iterable=True
>>
>> part...
> 
> 
> Can someone help me print a generator object?
> 

Have you tried Peter's suggestion?

If that didn't work (but what is that as_iterable argument for then?) just iterate over
the result and print each value as it comes out of the generator:

    print("Predictions:")
    for value in y:
        print(value)


Irmen



From bc at freeuk.com  Mon Dec  5 14:24:21 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 19:24:21 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.46.1480962939.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
Message-ID: <o24eqo$9mg$1@dont-email.me>

On 05/12/2016 18:34, eryk sun wrote:
> On Mon, Dec 5, 2016 at 3:41 PM, BartC <bc at freeuk.com> wrote:
>>
>> Are you saying that if someone executes:
>>
>>   subprocess.Popen(["python","a.py", "*"])
>>
>> the output will be: ['a.py','*']?
>>
>> In that case forget Windows vs. Linux, you now have a program that will get
>> command parameters processed differently depending on whether it was invoked
>> from a shell or not.
>>
>> Or a program that sometimes will see "*" as an argument, and sometimes a big
>> list of files that merges into all the other arguments.
>
> If it sees "*", it will try to open a file named "*".

And people still say that the way Windows works is crazy!

  That's a valid
> filename in Unix, but it should be avoided.

No, it should be prohibited, if the file API and/or shell assign special 
significance to it.

> I don't understand your concern regarding Unix shell scripting syntax.
> On Windows, are you as troubled that environment variables such as
> %USERNAME% get expanded by cmd.exe, but not by CreateProcess?

No, because I never use %USERNAME%, and it is obviously something 
associated with Windows' Batch language which I hardly ever use.

Not like file wildcards, which I use all the time, and don't associate 
them exclusively with a shell.

I also never expect any parameter that contains ? and * to be expanded 
then and there to whatever list of files there might happen to be. Maybe 
the files haven't been created yet, maybe the parameter has got NOTHING 
TO DO with filenames.

(For example, in Windows:

    >ren *.c *.d

Rename all .c files to .d files. None of the .d files exist (or, because 
of the point below, some isolated .d file exists). I wouldn't be able to 
emulate such a command under Linux, not without writing:

   rename *.c "*.d"

or even:

   rename "*.c" "*.d"

since, if the user forgets the second parameter, and there were two .c 
files, it would think I wanted to rename one .c file to another.)

But that leads to another weird behaviour I've just observed: if a 
parameter contains a filespec with wildcards, and there is at least one 
matching file, then you get a list of those files.

However, if there are no matching files, you get the filespec unchanged.

Now, what is a program supposed to with that? It seems it has to deal 
with wildcards after all. But also some behaviours can become erratic. 
This program asks for the name of a DAB radio station:

   >station ?LBC

Fine, the input is ?LBC (which is an actual station name where I live). 
But then at some point a file ALBC comes into existence; no connection. 
Now the same line gives the input ALBC, to perplex the user!

> Every command-line shell that I've ever used is also a quirky
> scripting language.

It seems shell language authors have nothing better to do than adding 
extra quirky features that sooner or later are going to bite somebody 
on the arse. Mainly I need a shell to help launch a program and give it 
some basic input; that's all.

(The people who think up obscure hot-key combinations belong in the same 
camp. I keep hitting key combinations that do weird things, then have to 
find a way out. One combination turned the display upside-down! Then I 
had to turn the monitor upside down while I investigated solutions.)

-- 
Bartc


From rosuav at gmail.com  Mon Dec  5 14:26:06 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 06:26:06 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <f970e3a2-1e49-a5d4-de90-1be2dc229ad4@gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o24a0n$m62$1@dont-email.me>
 <CAPTjJmqHOW6g3_Jg7KDgGgXKXRwSH7Yhm808dR9dL2yueZrkmw@mail.gmail.com>
 <f970e3a2-1e49-a5d4-de90-1be2dc229ad4@gmail.com>
Message-ID: <CAPTjJmo0fyTgZXZKeg5APrMAOYzd2b7D+nCAhXfY4_H4VD+bog@mail.gmail.com>

On Tue, Dec 6, 2016 at 6:08 AM, Michael Torrie <torriem at gmail.com> wrote:
> Agreed. I do this sort of trick all the time, even when I want to pass
> just a single file to a program.  I often use expansion for paths as well:
>
> somecommand /path/to/somelongname*withendpart/myepisode*2x03*mp4
>

"somecommand" is "vlc", isn't it :D

And I agree. I do this too. If you tab complete it, you can't as
easily use the up arrow. Compare:

vlc /video/Music/ASongToSingO/203*
vlc /video/Music/ASongToSingO/203\ Bugs\ Bunny\ at\ the\ Gaiety.ogg

The latter came from "203<tab>". Now, you want to play track 216
"Trial by Ruddy Patience" next. With the first one, it's up, left,
backspace two, patch in the new number, hit enter. With the second,
you have to backspace the whole title, then put in 216, and
tab-complete it again. Now, VLC on Windows is probably capable of
expanding globs, but why should it bother? Bash gives the exact same
result for both of those commands.

Plus, the Windows method depends on the globbing characters being
INVALID in file names. You absolutely cannot create a file called
"What's up, doc?.mkv" on Windows. And I'm not sure what other
characters are valid - are you allowed quotes? If so, how do you quote
the quote character? And what happens with network mounts?

With the Unix model, where the shell does the work, it's the shell's
job to provide an escaping mechanism. The shell doesn't have to be
externally consistent (although it helps a lot), but it can be
perfectly internally consistent. The *system* simply passes everything
along as-is, as an array of arguments.

Windows gives you less freedom than Unix does.

ChrisA


From torriem at gmail.com  Mon Dec  5 14:29:02 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 12:29:02 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24cr9$1ri$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
Message-ID: <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>

On 12/05/2016 11:50 AM, BartC wrote:
> It doesn't matter, and is not the concern of the shell. It should 
> restrict itself to the basic parsing that may be necessary when 
> parameters are separated by white-space and commas, if a parameter can 
> contain white-space or commas. That usually involves surrounding the 
> parameter with quotes.

That is just a difference of opinion and what you're used to.  A shell's
primary concern on my computer is providing a great environment for me
to work in, and for that, shell expansion is very important and useful
to me.  Also, in the Unix world (can't speak for Windows), we assemble
lots of little bash scripts into individual executable units, and string
them together. In other words, many "commands" we use in Unix are
actually little scripts themselves.  So having each and every little
script have to do its own glob expansion would be very tedious and
redundant. Moving that up to the shell makes an incredible amount of
sense given this structure.

> One would be very annoyed if, reading a CSV file, where each of N values 
> on a line correspond to a field of record, if one entry of "?LBC" 
> expanded itself to a dozen entries, screwing everything up.

I have no idea why you even put this in here.  What does reading a CSV
file have to do with glob expansion?  Sorry Bart, but you're grasping at
straws with that one.

> Shell command line processing shouldn't be attempting anything more than 
> that.

In your opinion.  Clearly there's a whole world out there that thinks
differently, and has their own good reasons.

> 
>> 	If they are command options, on a Linux shell options appear as
>>
>> X -A *.* -C -D
>>
>> Even Windows command processor separates optional/command stuff via
>>
>> X /A *.* /C /D
>>
>> ... or requires the parameter defining the file(s) to process as the last
>> arguments
> 
> Nonsense. If I pass that to a Python program that prints sys.argv, I get:
> 
> ['a.py', '/A', '*.*', '/C', '/D']
> 
> Presumably a Windows programmer is grown up enough to make their own 
> decisions as to what to do with that input.
> 
> All that's needed is one little standard library function to process 
> sys.argc, with unexpanded parameters, into a list of expanded arguments, 
> if that's really what someone wants (and then needing to trawl through 
> it all looking for the options).

Sure I understand that, and I can understand the appeal.  I just
disagree that it's this big issue you seem to think it is.

> Then you get the best of both worlds.
> 
> 
>> X A C D *.*
> 
> So how do I do:
> 
>     gcc *.c -lm
> 
> The -lm needs to go at the end.
> 
> Presumably it now needs to check each parameter seeing if it resembles a 
> file more than it does an option? And are options automatically taken 
> care of, or is that something that, unlike the easier wildcards, need to 
> be processed explicitly?

To add to what Chris said, all the different parameters have to be
looked at anyway, so I don't understand why it's such a burden to scan
them one by one and if they are a filename, add them to a list, if they
are a command-line argument, do something different.  And in the case of
gcc, one or more globs could appear, in any order.  So in Windows you'd
have to look at each argument anyway, and do n number of manual
expansions for non-argument parameters.  So this particular argument of
yours is rather weak.

Not having glob expansion be in individual programs makes things much
more explicit, from the program's point of view.  It provides an
interface that takes filename(s) and you provide them, either explicitly
(via popen with no shell), or you can do it implicitly but in an
interactive way via the shell using expansion.  Personal preference but
I believe it's a much better way because it's explicit from the
program's point of view as there's no question about the program's behavior.

Different strokes for different folks as they say.


From rosuav at gmail.com  Mon Dec  5 14:39:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 06:39:56 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24eqo$9mg$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me>
Message-ID: <CAPTjJmoQz8KvjXEYgp_3ZvhzkK6nzRcf5zz9Ro4Lv4YtUpYEwA@mail.gmail.com>

On Tue, Dec 6, 2016 at 6:24 AM, BartC <bc at freeuk.com> wrote:
>> If it sees "*", it will try to open a file named "*".
>
>
> And people still say that the way Windows works is crazy!
>
>  That's a valid
>>
>> filename in Unix, but it should be avoided.
>
>
> No, it should be prohibited, if the file API and/or shell assign special
> significance to it.
>

That implies that the file system, file API, and shell are all bound
together. You have to forbid in file names *any* character that has
*any* significance to *any* shell. That means no quotes, no asterisks
or question marks, no square brackets, no braces, etc, etc, etc.....
or, alternatively, restrict your shells to all perform the exact same
expansion - or no expansion at all, and then restrict all your
_applications_ to perform the same expansion.

Good luck.

ChrisA


From torriem at gmail.com  Mon Dec  5 14:48:11 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 12:48:11 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24eqo$9mg$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me>
Message-ID: <580898af-54c7-8494-4173-65be0b01d041@gmail.com>

Bored to day since it's -20 and I don't want to work outside.

On 12/05/2016 12:24 PM, BartC wrote:
>> If it sees "*", it will try to open a file named "*".
> 
> And people still say that the way Windows works is crazy!
> 
>   That's a valid
>> filename in Unix, but it should be avoided.
> 
> No, it should be prohibited, if the file API and/or shell assign special 
> significance to it.

Why? What does it matter to the file system what characters are in the
filename?

> 
>> I don't understand your concern regarding Unix shell scripting syntax.
>> On Windows, are you as troubled that environment variables such as
>> %USERNAME% get expanded by cmd.exe, but not by CreateProcess?
> 
> No, because I never use %USERNAME%, and it is obviously something 
> associated with Windows' Batch language which I hardly ever use.

cmd.exe *IS* the the "Windows' Batch" language.  So if you use cmd.exe,
you use this "Batch" language all the time.

> Not like file wildcards, which I use all the time, and don't associate 
> them exclusively with a shell.
>
> 
> I also never expect any parameter that contains ? and * to be expanded 
> then and there to whatever list of files there might happen to be. Maybe 
> the files haven't been created yet, maybe the parameter has got NOTHING 
> TO DO with filenames.
> 
> (For example, in Windows:
> 
>     >ren *.c *.d
> 
> Rename all .c files to .d files. None of the .d files exist (or, because 
> of the point below, some isolated .d file exists). I wouldn't be able to 
> emulate such a command under Linux, not without writing:
> 
>    rename *.c "*.d"
> 
> or even:
> 
>    rename "*.c" "*.d"

Wow. Does that actually work?  And work consistently?  How would it
handle globs like this:

renamae a*b??c*.c *.d

I can't think of any way to do that in a consistent way.  I can see it
working for the simple cases. Must have quite a bit of logic in rename
to handle all the corner cases.

Such a syntax, even with quoting, would be highly unusual to see in the
Unix world.  I think I'd rather just use a simple loop and have the
rename be explicit so I know it's working (and yes I do this all the
time) as I expect it to.

> since, if the user forgets the second parameter, and there were two .c 
> files, it would think I wanted to rename one .c file to another.)
> 
> But that leads to another weird behaviour I've just observed: if a 
> parameter contains a filespec with wildcards, and there is at least one 
> matching file, then you get a list of those files.
> 
> However, if there are no matching files, you get the filespec unchanged.

That's true for many shells (but not zsh by default which errors out).

> Now, what is a program supposed to with that? It seems it has to deal 
> with wildcards after all. But also some behaviours can become erratic. 
> This program asks for the name of a DAB radio station:

Again, it doesn't really matter.  If the program was looking for
filenames, the program will try to open the file as given ("a*.txt") and
if it succeeds it succeeds, if it fails it fails.  The program need not
care. Why should it?

> 
>    >station ?LBC
> 
> Fine, the input is ?LBC (which is an actual station name where I live). 
> But then at some point a file ALBC comes into existence; no connection. 
> Now the same line gives the input ALBC, to perplex the user!

Why would a file come into existence? I thought you said it refers to a
station identifier. No wonder you have such confused users if you're
creating files when you should be opening a URL or something.

I would expect in this case that the "station" program would return an
error, something like:

Error! Cannot find station "ALBC"

If station ids were supposed to start with a wildcard character, I'd
probably have to make a note in the help file that the station
identifiers have to be escaped or placed in quotes.  Or change my logic
to not require this ? character (if it was part of all identifiers we
can drop it safely).

> 
>> Every command-line shell that I've ever used is also a quirky
>> scripting language.
> 
> It seems shell language authors have nothing better to do than adding 
> extra quirky features that sooner or later are going to bite somebody 
> on the arse. Mainly I need a shell to help launch a program and give it 
> some basic input; that's all.

I think you'd quickly find that such a shell would be very non-useful.


From gordon at panix.com  Mon Dec  5 14:54:02 2016
From: gordon at panix.com (John Gordon)
Date: Mon, 5 Dec 2016 19:54:02 +0000 (UTC)
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me>
 <58449e18$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o230sk$8sa$1@dont-email.me>
 <58452645$0$1621$c3e8da3$5496439d@news.astraweb.com>
 <o249l7$knc$1@dont-email.me>
Message-ID: <o24gkq$j48$1@reader1.panix.com>

In <o249l7$knc$1 at dont-email.me> DFS <nospam at dfs.com> writes:

> > Shells don't just repeat the characters you type, they interpret them.

> Yes, I see that now.  I still don't think bash/shell should alter the 
> input going to another program.

But that's one of the reasons *to* use a shell!

  ls *.c                    ERROR No file named "*.c"
  ls > output               ERROR No file named ">"
  rm 'a file name'          ERROR No file named "'a file name'"
  cd ~/foo/bar              ERROR No file named "~/foo/bar"
  cat $HOME/file.txt        ERROR No file named '$HOME/file.txt' 
  vi $(grep -l foo *.txt)   ERROR No file named '$(grep -l foo *.txt)'

None of these commands would work if bash didn't "alter the input going to
another program".

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From random832 at fastmail.com  Mon Dec  5 15:09:47 2016
From: random832 at fastmail.com (Random832)
Date: Mon, 05 Dec 2016 15:09:47 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me>
 <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
Message-ID: <1480968587.3142679.809204449.2F5A49D0@webmail.messagingengine.com>

On Mon, Dec 5, 2016, at 14:48, Michael Torrie wrote:
> Wow. Does that actually work?  And work consistently?  How would it
> handle globs like this:

The rules are simpler than you're probably thinking of. There's actually
no relationship between globs on the left and on the right. Globs on the
left simply select the files to rename as normal, the glob pattern
doesn't inform the renaming operation.

A question mark on the right takes the character from *that character
position* in the original filename. That is, if you have "abc", "rename
ab? ?de" renames it to "ade", not "cde".

A star on the right takes all remaining characters. If you include a
dot, the "name" (everything before the last dot) and "extension" of the
file are considered separate components [so in adddition to rename *.c
*.d, renaming a.* b.* where you have a.py, a.pyc, a.pyo will work].

But if you don't include a dot, for example "rename abcde.fgh xyz*", it
will rename to xyzde.fgh .


From marko at pacujo.net  Mon Dec  5 15:14:17 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Mon, 05 Dec 2016 22:14:17 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <12703183-ca82-41ce-b1e2-61e75f86b71a@googlegroups.com>
 <CAPTjJmpx=O_CBcFsH9A_Uh6g209zw6FWvbn1RbmsmB2HQLNCLA@mail.gmail.com>
 <mailman.35.1480951854.2253.python-list@python.org>
 <877f7es4f5.fsf@elektro.pacujo.net>
 <CAPTjJmpMZuyc4m7kt6t+0AQuLRnduSaUzSHAgFxYBr1OL_AWGw@mail.gmail.com>
 <mailman.39.1480957578.2253.python-list@python.org>
 <87y3zuqoh8.fsf@elektro.pacujo.net>
 <CAPTjJmrb72_bFzCK_PdmV+8GzzAFWi25afmYerCouwrUkmo6rw@mail.gmail.com>
 <mailman.43.1480959318.2253.python-list@python.org>
Message-ID: <87shq2qgkm.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Tue, Dec 6, 2016 at 4:23 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Bash is nice, too nice. It makes it easy to write code that's riddled
>> with security holes. The glorious Unix tradition is to ignore the
>> pitfalls and forge ahead come what may.
>
> Bash assumes that the person typing commands has the full power to
> execute commands. I'm not sure what you mean by "security holes",
> unless it's passing text through bash that came from people who aren't
> allowed to type commands.

Let's mention a few traps:

 * Classic shell programming is built on the concept of string lists. A
   list of three file names could be stored in a variable as follows:

      files="a.txt b.txt c.txt"

   or maybe:

      files=$*

   Trouble is, whitespace is not a safe separator because it is a valid
   character in filenames.

   The only safe way out is to use bash's arrays, which are a deviation
   of the classic foundation. That's why most (?) shell scripts just
   ignore the possibility of whitespace in filenames.

 * A common, classic pattern to pipeline data is something like this:

      find_files |
      while read filename; do
          ... do a thing or two with "$filename" ...
      done

   Unfortunately, the pattern suffers from several problems:

    1. A filename can legally contain a newline.

    2. A filename can legally begin and end with whitespace which is
       stripped by the "read" builtin.

    3. The "read" builtin treats the backslash as an escape character
       unless the "-r" option is specified. The backslash is a valid
       character in a filename.

 * The classic "test" builtin (or "[ ... ]") has ambigous syntax:

      if [ -e "$filename" -o -e "$filename.bak" ]; then
          ...
      fi

   results in a syntax error if "$filename" should be "=".

 * This fails gracefully:

      local x
      x=$(false) || exit

   This doesn't fail at all:

      local x=$(false) || exit


Marko


From torriem at gmail.com  Mon Dec  5 15:34:08 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 13:34:08 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <1480968587.3142679.809204449.2F5A49D0@webmail.messagingengine.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me> <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
 <1480968587.3142679.809204449.2F5A49D0@webmail.messagingengine.com>
Message-ID: <87108f09-8e1d-edc8-325b-6c3579f88ef9@gmail.com>

On 12/05/2016 01:09 PM, Random832 wrote:
> The rules are simpler than you're probably thinking of. There's actually
> no relationship between globs on the left and on the right. Globs on the
> left simply select the files to rename as normal, the glob pattern
> doesn't inform the renaming operation.
> 
> A question mark on the right takes the character from *that character
> position* in the original filename. That is, if you have "abc", "rename
> ab? ?de" renames it to "ade", not "cde".
> 
> A star on the right takes all remaining characters. If you include a
> dot, the "name" (everything before the last dot) and "extension" of the
> file are considered separate components [so in adddition to rename *.c
> *.d, renaming a.* b.* where you have a.py, a.pyc, a.pyo will work].
> 
> But if you don't include a dot, for example "rename abcde.fgh xyz*", it
> will rename to xyzde.fgh .

Ahh.  Good explanation.  Thank you.  So in the case of rename, the
second argument isn't actually a glob at all (even though it looks like
one).  A useful simplification to be sure, though not one I'd want to
emulate (it relies on Windows' idea of file extensions to really work
correctly).


From bc at freeuk.com  Mon Dec  5 15:35:10 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 20:35:10 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.54.1480967297.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me> <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
 <mailman.54.1480967297.2253.python-list@python.org>
Message-ID: <o24ivi$q35$1@dont-email.me>

On 05/12/2016 19:48, Michael Torrie wrote:
> Bored to day since it's -20 and I don't want to work outside.
>
> On 12/05/2016 12:24 PM, BartC wrote:

>> (For example, in Windows:
>>
>>     >ren *.c *.d
>>
>> Rename all .c files to .d files. None of the .d files exist (or, because
>> of the point below, some isolated .d file exists). I wouldn't be able to
>> emulate such a command under Linux, not without writing:
>>
>>    rename *.c "*.d"
>>
>> or even:
>>
>>    rename "*.c" "*.d"
>
> Wow. Does that actually work?  And work consistently?  How would it
> handle globs like this:
>
> renamae a*b??c*.c *.d

That's not really the point, which is that a parameter may use * and ? 
which, even if they are related to file names, may not refer to actual 
input files in this directory. Here, they are output files. But this is 
just one of dozens of reasons why automatic expansion of such patterns 
is not desirable.

(And yes sometimes it works, for simple cases, and sometimes it screws 
up. But it is up to the program how it defines its input and what it 
will do with it. It shouldn't be up to the shell to expand such names in 
ALL circumstances whether that is meaningful to the app or not.)

>> But that leads to another weird behaviour I've just observed: if a
>> parameter contains a filespec with wildcards, and there is at least one
>> matching file, then you get a list of those files.
>>
>> However, if there are no matching files, you get the filespec unchanged.
>
> That's true for many shells (but not zsh by default which errors out).
>
>> Now, what is a program supposed to with that? It seems it has to deal
>> with wildcards after all. But also some behaviours can become erratic.
>> This program asks for the name of a DAB radio station:
>
> Again, it doesn't really matter.  If the program was looking for
> filenames, the program will try to open the file as given ("a*.txt") and
> if it succeeds it succeeds, if it fails it fails.  The program need not
> care. Why should it?

Really, the program should just fail? It seems to me that the program is 
then obliged to deal with wildcards after all, even if just to detect 
that wildcards have been used and that is reason for the error. But my 
example was how that behaviour can stop programs working in random ways.

>> Fine, the input is ?LBC (which is an actual station name where I live).
>> But then at some point a file ALBC comes into existence; no connection.
>> Now the same line gives the input ALBC, to perplex the user!
>
> Why would a file come into existence? I thought you said it refers to a
> station identifier. No wonder you have such confused users if you're
> creating files when you should be opening a URL or something.

The file is irrelevant. It's just a bit of junk, or something that is 
used for some other, unrelated purpose.

The important thing is that creation of that file FOR ANY REASON could 
screw up a program which is asking for input that is unrelated to any files.

> I would expect in this case that the "station" program would return an
> error, something like:
>
> Error! Cannot find station "ALBC"

And that's the error! But how is the user supposed to get round it? It's 
an error due entirely to unconditional shell expansion of any parameter 
that looks like it might be a filename with wildcards.

> If station ids were supposed to start with a wildcard character, I'd
> probably have to make a note in the help file that the station
> identifiers have to be escaped or placed in quotes.

I've no idea why that station starts with "?" (it appears on the station 
list, but have never tried to listen to it). But this is just an example 
of input that might use ? or * that has nothing to do files, yet can be 
affected by the existence of some arbitrary file.

And somebody said doing it this way is more secure than how Windows does 
things!

>> It seems shell language authors have nothing better to do than adding
>> extra quirky features that sooner or later are going to bite somebody
>> on the arse. Mainly I need a shell to help launch a program and give it
>> some basic input; that's all.
>
> I think you'd quickly find that such a shell would be very non-useful.

For about five minutes, until someone produces an add-on with the 
missing features.

-- 
Bartc


From bc at freeuk.com  Mon Dec  5 15:55:41 2016
From: bc at freeuk.com (BartC)
Date: Mon, 5 Dec 2016 20:55:41 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.52.1480966148.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
Message-ID: <o24k60$ufp$1@dont-email.me>

On 05/12/2016 19:29, Michael Torrie wrote:
> On 12/05/2016 11:50 AM, BartC wrote:

>> So how do I do:
>>
>>     gcc *.c -lm
>>
>> The -lm needs to go at the end.
>>
>> Presumably it now needs to check each parameter seeing if it resembles a
>> file more than it does an option? And are options automatically taken
>> care of, or is that something that, unlike the easier wildcards, need to
>> be processed explicitly?

This was a response to someone saying the wildcard param needs to be at 
the end. There need be no such restriction if handled properly (ie. no 
auto-expansion).

But a similar example, suppose a syntax is:

   appl *.* [options]

but one of the files in the list is called "-lm", or some other option 
that can do things the user didn't want (with gcc, -lm is harmless).

Without expansion, input is easy to parse: filespec, followed by 
optional options. But with expansion, now you have to decide if a 
particular argument is an option, or a filename.

And if there's an error in an option, you may have to abort, which means 
throwing away that list of files which, in some cases, can run into 
millions.

> Not having glob expansion be in individual programs makes things much
> more explicit, from the program's point of view.  It provides an
> interface that takes filename(s) and you provide them, either explicitly
> (via popen with no shell), or you can do it implicitly but in an
> interactive way via the shell using expansion.  Personal preference but
> I believe it's a much better way because it's explicit from the
> program's point of view as there's no question about the program's behavior.
>
> Different strokes for different folks as they say.

I must have given ten or twenty scenarios where such auto-expansion is 
problematical. And yet people are still in denial. It's been in Unix for 
decades therefore there's nothing wrong with it!

But maybe people simply avoid all the situations that cause problems. 
Interfaces are specified in a certain manner, given that input can be 
any long stream of filenames and/or options with no order and no 
positional significance. Non-file parameters that use ? or * are 
prohibited. You can't do the equivalent of:

  >DIR *.b *.c

And get a list of *.b files, with a heading and summary lines, then a 
list of *.c files with its own heading and summary. It would be one 
monolithic list.

So everyone is working around the restrictions and the problems. Which 
is exactly what I would have to do.

That doesn't change the fact that the Windows approach is considerably 
more flexible and allowing more possibilities.

-- 
Bartc




From best_lay at yahoo.com  Mon Dec  5 15:58:56 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 14:58:56 -0600
Subject: Detect Linux Runlevel
Message-ID: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>

I there a way to detect what the Linux runlevel is from
within a Python program?  I would like to be able to do
it without the use of an external program such as 'who'
or 'runlevel'.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Mon Dec  5 16:35:29 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 15:35:29 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
Message-ID: <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>

On Mon, 05 Dec 2016 16:25:56 -0500, DFS wrote:

> On 12/05/2016 03:58 PM, Wildman wrote:
>> I there a way to detect what the Linux runlevel is from
>> within a Python program?  I would like to be able to do
>> it without the use of an external program such as 'who'
>> or 'runlevel'.
> 
> 
> Why not?
> 
> '>>> import os
> '>>> os.system("systemctl get-default")
> graphical.target
> 
> 
> 
> systemd 'graphical.target' corresponds to the old runlevel 5.

Thanks but I knew about systemctl.  As I already said my
goal is to do it without the use of an external program.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From lew.pitcher at digitalfreehold.ca  Mon Dec  5 16:38:00 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Mon, 05 Dec 2016 16:38:00 -0500
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
Message-ID: <o24ml9$74d$1@dont-email.me>

On Monday December 5 2016 16:25, in comp.lang.python, "DFS" <nospam at dfs.com>
wrote:

> On 12/05/2016 03:58 PM, Wildman wrote:
>> I there a way to detect what the Linux runlevel is from
>> within a Python program?  I would like to be able to do
>> it without the use of an external program such as 'who'
>> or 'runlevel'.
> 
> 
> Why not?
> 
> '>>> import os
> '>>> os.system("systemctl get-default")
> graphical.target

Because

  $ cat rlevel.py
  import os
  os.system("systemctl get-default")
  16:36 $ python rlevel.py
  sh: systemctl: command not found
  16:36 $ 


> systemd 'graphical.target' corresponds to the old runlevel 5.

Yes? So?

The OP asked for the runlevel, not the systemd target.


-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From marko at pacujo.net  Mon Dec  5 16:59:48 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Mon, 05 Dec 2016 23:59:48 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
Message-ID: <87h96iqbor.fsf@elektro.pacujo.net>

Wildman <best_lay at yahoo.com>:
> Thanks but I knew about systemctl. As I already said my goal is to do
> it without the use of an external program.

Inspect:

   <URL:
   https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c>

In particular:

   get_state_one_unit()

Then, proceed to:

   <URL: https://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html>


Marko


From python.list at tim.thechases.com  Mon Dec  5 17:08:57 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Mon, 5 Dec 2016 16:08:57 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
Message-ID: <20161205160857.70bff64d@bigbox.christie.dr>

On 2016-12-05 14:58, Wildman via Python-list wrote:
> I there a way to detect what the Linux runlevel is from
> within a Python program?  I would like to be able to do
> it without the use of an external program such as 'who'
> or 'runlevel'.

You can use something like

https://gist.github.com/likexian/f9da722585036d372dca

to parse the /var/run/utmp contents.  Based on some source-code
scrounging, it looks like you want the first field to be "1" for the
"runlevel" account.  To extract the actual runlevel, you can take
the PID value from the second column ("53" in my example here) and
take it's integer value mod 256 (AKA "& 0xff") to get the character
value.  So chr(int("53") & 0xff) returns "5" in my case, which is my
runlevel.

Additional links I found helpful while searching:

https://casper.berkeley.edu/svn/trunk/roach/sw/busybox-1.10.1/miscutils/runlevel.c
https://github.com/garabik/python-utmp

-tkc





From rosuav at gmail.com  Mon Dec  5 17:24:45 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 09:24:45 +1100
Subject: Detect Linux Runlevel
In-Reply-To: <o24ml9$74d$1@dont-email.me>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
Message-ID: <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>

On Tue, Dec 6, 2016 at 8:38 AM, Lew Pitcher
<lew.pitcher at digitalfreehold.ca> wrote:
> The OP asked for the runlevel, not the systemd target.

Runlevels don't exist in systemd. And systemd targets don't exist in
Upstart. The question "what runlevel are we in" does not make sense
unless you're using an init system that works on the basis of
runlevels (eg sysvinit).

ChrisA


From marko at pacujo.net  Mon Dec  5 17:29:58 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 06 Dec 2016 00:29:58 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
Message-ID: <87d1h6qaah.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Tue, Dec 6, 2016 at 8:38 AM, Lew Pitcher
> <lew.pitcher at digitalfreehold.ca> wrote:
>> The OP asked for the runlevel, not the systemd target.
>
> Runlevels don't exist in systemd. And systemd targets don't exist in
> Upstart. The question "what runlevel are we in" does not make sense
> unless you're using an init system that works on the basis of
> runlevels (eg sysvinit).

In fact, systemd is not an init system for Linux. Linux is the kernel of
the systemd operating system. Systemd is the

      One Ring to rule them all, One Ring to find them,
   One Ring to bring them all and in the darkness bind them


Marko


From best_lay at yahoo.com  Mon Dec  5 17:34:33 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 16:34:33 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
Message-ID: <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>

On Mon, 05 Dec 2016 23:59:48 +0200, Marko Rauhamaa wrote:

> Wildman <best_lay at yahoo.com>:
>> Thanks but I knew about systemctl. As I already said my goal is to do
>> it without the use of an external program.
> 
> Inspect:
> 
>    <URL:
>    https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c>
> 
> In particular:
> 
>    get_state_one_unit()

Too bad I don't speak C.  I am an amateur programmer and most or all
my experience has been with assembly and various flavors of BASIC,
including VB and PowerBASIC.  I did look over the code but I guess
I'm just a rebel without a clue. <g>

> Then, proceed to:
> 
>    <URL: https://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html>
> 
> 
> Marko

Even if I could ?figure this out, I can't depend on systemd being
installed.  The way things are going that may change tho.  I have
bookmarked the pages for possible future use.

Thank you, I do appreciate your post.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From torriem at gmail.com  Mon Dec  5 17:38:26 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 15:38:26 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24ivi$q35$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me> <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
 <mailman.54.1480967297.2253.python-list@python.org>
 <o24ivi$q35$1@dont-email.me>
Message-ID: <daadf782-ff2d-5b3c-0127-879f138be947@gmail.com>

On 12/05/2016 01:35 PM, BartC wrote:
>>> It seems shell language authors have nothing better to do than adding
>>> extra quirky features that sooner or later are going to bite somebody
>>> on the arse. Mainly I need a shell to help launch a program and give it
>>> some basic input; that's all.
>>
>> I think you'd quickly find that such a shell would be very non-useful.
> 
> For about five minutes, until someone produces an add-on with the 
> missing features.

Which will re-implement everything you stripped out. :)

As I've gotten older I've learned the truth of this quotation:
"Those who do not understand UNIX are condemned to reinvent it, poorly."
-- Henry Spencer




From torriem at gmail.com  Mon Dec  5 17:39:24 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 15:39:24 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
Message-ID: <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>

On 12/05/2016 03:34 PM, Wildman via Python-list wrote:
> Too bad I don't speak C.  I am an amateur programmer and most or all
> my experience has been with assembly and various flavors of BASIC,
> including VB and PowerBASIC.  I did look over the code but I guess
> I'm just a rebel without a clue. <g>

May I ask what you are trying to accomplish?  Why does the runlevel matter?


From torriem at gmail.com  Mon Dec  5 17:44:54 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 15:44:54 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <87d1h6qaah.fsf@elektro.pacujo.net>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
Message-ID: <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>

On 12/05/2016 03:29 PM, Marko Rauhamaa wrote:
> Chris Angelico <rosuav at gmail.com>:
> 
>> On Tue, Dec 6, 2016 at 8:38 AM, Lew Pitcher
>> <lew.pitcher at digitalfreehold.ca> wrote:
>>> The OP asked for the runlevel, not the systemd target.
>>
>> Runlevels don't exist in systemd. And systemd targets don't exist in
>> Upstart. The question "what runlevel are we in" does not make sense
>> unless you're using an init system that works on the basis of
>> runlevels (eg sysvinit).
> 
> In fact, systemd is not an init system for Linux. Linux is the kernel of
> the systemd operating system. Systemd is the
> 
>       One Ring to rule them all, One Ring to find them,
>    One Ring to bring them all and in the darkness bind them

Well I for one am glad of the systemd init system inside of my Linux
operating system.  I would far rather deal with service ini files than
long arcane bash scripts that often re-implement (poorly) things like
pid files, and attempts to prevent more than one instance from running.
You were the one that posted earlier today about the many perils of
programming complicated scripts in bash. Init scripts pretty much hit on
all of those gotchas!  It boggles my mind that people would actually
argue in favor of maintaining such things (and yes I have done it on
many servers over the years and it was always a pain to deal with custom
daemons).

I don't use very many of the systemd modules (most are not even
installed on my machine), nor will I ever need them. In fact I still
have rsyslog running.


From lew.pitcher at digitalfreehold.ca  Mon Dec  5 17:45:57 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Mon, 05 Dec 2016 17:45:57 -0500
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
Message-ID: <o24qkm$lmo$1@dont-email.me>

On Monday December 5 2016 17:24, in comp.lang.python, "Chris Angelico"
<rosuav at gmail.com> wrote:

> On Tue, Dec 6, 2016 at 8:38 AM, Lew Pitcher
> <lew.pitcher at digitalfreehold.ca> wrote:
>> The OP asked for the runlevel, not the systemd target.
> 
> Runlevels don't exist in systemd. And systemd targets don't exist in
> Upstart. The question "what runlevel are we in" does not make sense
> unless you're using an init system that works on the basis of
> runlevels (eg sysvinit).

I repeat: The OP asked for the runlevel, not the systemd target.

That should tell you that an answer involving systemd "does not make sense".

To the OP: as others have said, the file
  /var/run/utmp
contains various records, including the RUN_LVL (runlevel) record. You can
find some documentation in utmp(3), including a record layout, and an values
list.

I don't know that python includes a standard or builtin method to parse the
utmp file; to retrieve the runlevel, you may have to code your own access
routine as part of your python code, or resort to invoking an external
program like who(1).

HTH
-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From steve+python at pearwood.info  Mon Dec  5 17:49:50 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 06 Dec 2016 09:49:50 +1100
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
References: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
 <o0ifbb$mc9$1@blaine.gmane.org>
 <mailman.171.1479327928.30068.python-list@python.org>
 <fafad656-6c14-4714-b810-4c21674cbcf2@googlegroups.com>
Message-ID: <5845ef10$0$1586$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 05:39 am, vmahajan at centerpointmedia.com wrote:

> Can someone help me print a generator object?

The same way as you print any other object:

print(obj)  # Python 3

print obj  # Python 2


But what you're actually asking for is a way to print the values produced by
the generator object. You do that the same way as you would print the
values produced by any sequence or iterable. You can print one value per
line:

for value in obj:
    print(value)


Or you can convert to a list, and then print the list:

print(list(obj))


Or you can format it yourself, any way you like, by writing some code:


display_string = '::'.join(str(value).upper() for value in obj)
print('[[' + display_string + ']]')




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From eryksun at gmail.com  Mon Dec  5 18:09:46 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 5 Dec 2016 23:09:46 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>

On Mon, Dec 5, 2016 at 4:49 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
>
> You've never used cmd.com or command.exe? "The DOS prompt"?

The default Windows shell is "cmd.exe", and it's informally called the
"Command Prompt", not "DOS Prompt". In Windows 9x it was accurate to
say DOS prompt, since the shell was COMMAND.COM, which used DOS system
calls. But that branch of Windows has been dead for over a decade.

> Even the DOS prompt supports some level of globbing. Its been a while since
> I've used the DOS prompt in anger, but I seem to recall being able to do
> things like:
>
> dir a*

"dir" is a built-in command. It calls FindFirstFileExW to list each
directory that it walks over.

FindFirstFileExW converts the glob to a form supported by the system
call NtQueryDirectoryFile. In this case that's simply "a*". In other
cases it tries to match the behavior of MS-DOS globbing, which
requires rewriting the pattern to use DOS_STAR ('<'), DOS_QM ('>'),
and DOS_DOT ('"'). To simplify the implementation, the five wildcard
characters are reserved. They're not allowed in filenames.

The I/O manager leaves the implementation up to the filesystem driver.
NtQueryDirectoryFile gets dispatched to the driver as an I/O request
packet with major function IRP_MJ_DIRECTORY_CONTROL and minor function
IRP_MN_QUERY_DIRECTORY. The driver does most of the work, including
filtering the directory listing by the FileName argument. But a driver
writer doesn't have to reinvent the wheel here; the filesystem runtime
library has string comparison functions that support wildcards, such
as FsRtlIsNameInExpression.

> *every single command and application* has to re-implement its own globbing,
> very possibly inconsistently.

C/C++ programs can link with wsetargv.obj to support command-line globbing.

For Python, this dependency can be added in PCBuild/python.vcxproj in
the linker configuration:

    <Link>
      <SubSystem>Console</SubSystem>
      <StackReserveSize>2000000</StackReserveSize>
      <BaseAddress>0x1d000000</BaseAddress>
      <AdditionalDependencies>wsetargv.obj</AdditionalDependencies>
    </Link>

For example:

    C:\Temp\test>dir /b
    a.txt
    b.dat
    c.bin

    C:\Temp\test>python -c "import sys;print(sys.argv)" *.txt *.dat
    ['-c', 'a.txt', 'b.dat']


From best_lay at yahoo.com  Mon Dec  5 18:12:34 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 17:12:34 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
Message-ID: <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>

On Mon, 05 Dec 2016 15:39:24 -0700, Michael Torrie wrote:

> On 12/05/2016 03:34 PM, Wildman via Python-list wrote:
>> Too bad I don't speak C.  I am an amateur programmer and most or all
>> my experience has been with assembly and various flavors of BASIC,
>> including VB and PowerBASIC.  I did look over the code but I guess
>> I'm just a rebel without a clue. <g>
> 
> May I ask what you are trying to accomplish?  Why does the runlevel matter?

Of course you may.  It is part of a program I'm working on
that reports different information about the Linux system it
is running on.  A program similar to inxi.  And I am trying
to write it without using external programs, where possible.

I am a hobby programmer and I've been trying to learn python
for a few months now.  The program is 'worthlessware' but it
is a 'learning experience' for me.  A friend wrote a similar
program in Bash script and I have been working on translating
it to Python.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From marko at pacujo.net  Mon Dec  5 18:37:41 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 06 Dec 2016 01:37:41 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
Message-ID: <878truq75m.fsf@elektro.pacujo.net>

Michael Torrie <torriem at gmail.com>:

> On 12/05/2016 03:29 PM, Marko Rauhamaa wrote:
>> In fact, systemd is not an init system for Linux. Linux is the kernel of
>> the systemd operating system. Systemd is the
>> 
>>       One Ring to rule them all, One Ring to find them,
>>    One Ring to bring them all and in the darkness bind them
>
> Well I for one am glad of the systemd init system inside of my Linux
> operating system. I would far rather deal with service ini files than
> long arcane bash scripts that often re-implement (poorly) things like
> pid files, and attempts to prevent more than one instance from
> running.

The situation before systemd *was* atrocious. Most of the protests
against systemd were misplaced. They advocated the ancient hacker
culture that placed the supreme authority on the System Administrator,
who fashioned the system into his own image.

I appreciate that finally there was a bold soul who took the point of
view of an Architect and laid down some higher-level principles for the
whole system to abide by.

Unfortunately, I am not wholly impressed by the end result. Mogadishu
has been replaced by Pyongyang. Some age-old Unix principles have been
abandoned without clear justification. For example, I was appalled to
find out that a systemd unit can be configured to react on the exit
status of a child process of a daemon.

Also, now D-Bus is a fixture for every daemon writer.

> You were the one that posted earlier today about the many perils of
> programming complicated scripts in bash. Init scripts pretty much hit on
> all of those gotchas!

Guess what -- the unit files come with their own gotchas. For example,
there is no simple way to convert an arbitrary pathname into an
ExecStart entry of a unit file.

The .ini file format was a lousy choice. Why not choose JSON in this day
and age?

Yes, the SysV init system had become a jungle. Still, I wish it had been
replaced with a rigorous protocol. The units should have been programs
that comply with given contracts.


Marko


From marko at pacujo.net  Mon Dec  5 18:38:25 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 06 Dec 2016 01:38:25 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me>
 <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
 <mailman.54.1480967297.2253.python-list@python.org>
 <o24ivi$q35$1@dont-email.me>
 <daadf782-ff2d-5b3c-0127-879f138be947@gmail.com>
 <mailman.59.1480977511.2253.python-list@python.org>
Message-ID: <874m2iq74e.fsf@elektro.pacujo.net>

Michael Torrie <torriem at gmail.com>:
> As I've gotten older I've learned the truth of this quotation:
> "Those who do not understand UNIX are condemned to reinvent it, poorly."
> -- Henry Spencer

I thought you kinda liked systemd...


Marko


From torriem at gmail.com  Mon Dec  5 19:08:25 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 17:08:25 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <874m2iq74e.fsf@elektro.pacujo.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <CAPTjJmrtN58Ci2xcT6EsYgfBec2NFfDrAwc1+3xQ72xEn68zVg@mail.gmail.com>
 <mailman.30.1480940611.2253.python-list@python.org>
 <o23sfi$1bn$1@dont-email.me>
 <CAPTjJmpuWm3P=+EZ9Cw36in3W3YF=P+apy8p51DPjDbN5_2mFw@mail.gmail.com>
 <mailman.34.1480950353.2253.python-list@python.org>
 <o241pc$l69$1@dont-email.me>
 <CACL+1asq00-bqmqNEePZiO98U2y6G-UmYofJ8ufYiPtk6YEvAA@mail.gmail.com>
 <mailman.46.1480962939.2253.python-list@python.org>
 <o24eqo$9mg$1@dont-email.me> <580898af-54c7-8494-4173-65be0b01d041@gmail.com>
 <mailman.54.1480967297.2253.python-list@python.org>
 <o24ivi$q35$1@dont-email.me> <daadf782-ff2d-5b3c-0127-879f138be947@gmail.com>
 <mailman.59.1480977511.2253.python-list@python.org>
 <874m2iq74e.fsf@elektro.pacujo.net>
Message-ID: <4db1fdfe-d747-d2f8-f935-cd73c37b4f8d@gmail.com>

On 12/05/2016 04:38 PM, Marko Rauhamaa wrote:
> Michael Torrie <torriem at gmail.com>:
>> As I've gotten older I've learned the truth of this quotation:
>> "Those who do not understand UNIX are condemned to reinvent it, poorly."
>> -- Henry Spencer
> 
> I thought you kinda liked systemd...

Yup I do.


From torriem at gmail.com  Mon Dec  5 19:08:44 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 17:08:44 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <878truq75m.fsf@elektro.pacujo.net>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
 <878truq75m.fsf@elektro.pacujo.net>
Message-ID: <f9ea9c78-fcdc-d909-5c4f-732865cf34cf@gmail.com>

On 12/05/2016 04:37 PM, Marko Rauhamaa wrote:
> Unfortunately, I am not wholly impressed by the end result. Mogadishu
> has been replaced by Pyongyang. Some age-old Unix principles have been
> abandoned without clear justification. For example, I was appalled to
> find out that a systemd unit can be configured to react on the exit
> status of a child process of a daemon.

I have yet to see any evidence of this Pyonguang situation.

What is wrong with reacting to the exit status of a daemon's child process?

> Also, now D-Bus is a fixture for every daemon writer.

But not directly. It need not be a dependency on the part of the daemon
writer. In fact the daemon writer may now leave out deamonizing code
entirely if he or she wishes.  The systemd api is entirely optional for
the daemon to use.

> Guess what -- the unit files come with their own gotchas. For example,
> there is no simple way to convert an arbitrary pathname into an
> ExecStart entry of a unit file.
> 
> The .ini file format was a lousy choice. Why not choose JSON in this day
> and age?

But why json?  ini files are at least fairly human write-able and
read-able. Json is great, but not for this application. Thank goodness
they didn't choose xml. I never liked working with LaunchDaemon's plist
files or Solaris' service definition files.

> Yes, the SysV init system had become a jungle. Still, I wish it had been
> replaced with a rigorous protocol. The units should have been programs
> that comply with given contracts.

Yes there is certainly merit to this idea, if anyone could agree on a
protocol.



From Bernd.Nawothnig at t-online.de  Mon Dec  5 19:14:35 2016
From: Bernd.Nawothnig at t-online.de (Bernd Nawothnig)
Date: Tue, 6 Dec 2016 01:14:35 +0100
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
Message-ID: <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>

On 2016-12-05, Wildman wrote:
> And I am trying to write it without using external programs, where
> possible.

That is not the Unix way.

> I am a hobby programmer and I've been trying to learn python
> for a few months now.  The program is 'worthlessware' but it
> is a 'learning experience' for me.  

It looks for me like a worthless learning experience.

> A friend wrote a similar program in Bash script and I have been
> working on translating it to Python.

Stay with shell script for such tasks. It is never a good idea to
choose the programming language before closer evaluating the problem.




Bernd

-- 
No time toulouse


From best_lay at yahoo.com  Mon Dec  5 19:26:51 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 18:26:51 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
Message-ID: <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>

On Mon, 05 Dec 2016 16:08:57 -0600, Tim Chase wrote:

> On 2016-12-05 14:58, Wildman via Python-list wrote:
>> I there a way to detect what the Linux runlevel is from
>> within a Python program?  I would like to be able to do
>> it without the use of an external program such as 'who'
>> or 'runlevel'.
> 
> You can use something like
> 
> https://gist.github.com/likexian/f9da722585036d372dca
> 
> to parse the /var/run/utmp contents.  Based on some source-code
> scrounging, it looks like you want the first field to be "1" for the
> "runlevel" account.  To extract the actual runlevel, you can take
> the PID value from the second column ("53" in my example here) and
> take it's integer value mod 256 (AKA "& 0xff") to get the character
> value.  So chr(int("53") & 0xff) returns "5" in my case, which is my
> runlevel.
> 
> Additional links I found helpful while searching:
> 
> https://casper.berkeley.edu/svn/trunk/roach/sw/busybox-1.10.1/miscutils/runlevel.c
> https://github.com/garabik/python-utmp
> 
> -tkc

That is exactly the kind of thing I was looking for.  Thank you.
Now all I have to do is get it to work with Python3.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From clvanwall at gmail.com  Mon Dec  5 20:06:49 2016
From: clvanwall at gmail.com (clvanwall)
Date: Mon, 05 Dec 2016 19:06:49 -0600
Subject: When will they fix Python _dbm?
Message-ID: <w4o957e3n2v24639mwec7vrj.1480986370390@email.android.com>

will thid do? ?John


Sent from my Galaxy Tab? A
-------- Original message --------From: justin walters <walters.justin01 at gmail.com> Date: 12/5/16  11:13 AM  (GMT-06:00) To: python-list at python.org Subject: Re: When will they fix Python _dbm? 
On Mon, Dec 5, 2016 at 6:45 AM, clvanwall <clvanwall at gmail.com> wrote:

> I have been a Perl programmer for 15+ years and decided to give Python a
> try.? My platform is windows and I installed the latest 3.5.2. Next I
> decided to convert a perl program that uses a ndbm database since according
> to the doc on python, it should be able to work with it.? Needless to say,
> I get: dbm.error: db type is dbm.ndbm, but the module is not available
> Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!
> That's a long time for a bug to be left open.
> John Van Walleghen
>
>
> Sent from my Galaxy Tab? A
> --
> https://mail.python.org/mailman/listinfo/python-list
>


Hi there,

Could you please post code that is giving you an error?
-- 
https://mail.python.org/mailman/listinfo/python-list

From torriem at gmail.com  Mon Dec  5 20:25:58 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 18:25:58 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
Message-ID: <9e4abd0f-933e-981b-527e-2f019460aac7@gmail.com>

On 12/05/2016 05:14 PM, Bernd Nawothnig wrote:
> On 2016-12-05, Wildman wrote:
>> And I am trying to write it without using external programs, where
>> possible.
> 
> That is not the Unix way.
> 
>> I am a hobby programmer and I've been trying to learn python
>> for a few months now.  The program is 'worthlessware' but it
>> is a 'learning experience' for me.  
> 
> It looks for me like a worthless learning experience.
> 
>> A friend wrote a similar program in Bash script and I have been
>> working on translating it to Python.
> 
> Stay with shell script for such tasks. It is never a good idea to
> choose the programming language before closer evaluating the problem.I

I think Python is a good choice for such a utility, but I agree it is
much better to rely on these external utilities as children to do the
platform-dependent work, rather than try to re-implement everything in
Python.  A long time ago I wrote a simple wrapper to Popen that would
run a command and return the standard out and standard error to me.


From nathan.ernst at gmail.com  Mon Dec  5 21:42:19 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Mon, 5 Dec 2016 20:42:19 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me>
 <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
Message-ID: <CAPjBxg-KduiMmmcD0i8sHPiuN1mBQX6As00RmJHYf=2FaU+SPw@mail.gmail.com>

Rather than argue about what is/should be allowed by a filesystem, this
defines what is allowed on NTFS (default for modern Windows systems):
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

One can complain about whether or not something should be allowed, but,
you'd have to take that up with Microsoft (and I'll doubt you'll make a
convincing enough argument for them to change it).

What is allowed on linux may be defined by linux itself, and it may be
restricted by the filesystem type itself (I don't know).

Regards,
Nathan

On Mon, Dec 5, 2016 at 8:25 PM, Dennis Lee Bieber <wlfraed at ix.netcom.com>
wrote:

> On Mon, 5 Dec 2016 20:55:41 +0000, BartC <bc at freeuk.com> declaimed the
> following:
>
> >This was a response to someone saying the wildcard param needs to be at
> >the end. There need be no such restriction if handled properly (ie. no
> >auto-expansion).
> >
>         That only applies if there is no prefix indicating a command option
> from a file name.
>
> >but one of the files in the list is called "-lm", or some other option
>
>         -lm is not a valid file name on the OS's that use - as an option
> prefix.
>
> >Without expansion, input is easy to parse: filespec, followed by
> >optional options. But with expansion, now you have to decide if a
> >particular argument is an option, or a filename.
> >
>         And you do that using a delimiter character that is not valid in
> filenames. On Windows, file names can not have a /, and that is the
> character used by the command line interpreter to indicate an option
> follows. On UNIX, - is used as the delimiter of an option.
>
> --
>         Wulfraed                 Dennis Lee Bieber         AF6VN
>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From steve+python at pearwood.info  Mon Dec  5 21:44:14 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 06 Dec 2016 13:44:14 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>
 <mailman.62.1480979436.2253.python-list@python.org>
Message-ID: <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 10:09 am, eryk sun wrote:

> On Mon, Dec 5, 2016 at 4:49 PM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>>
>> You've never used cmd.com or command.exe? "The DOS prompt"?
> 
> The default Windows shell is "cmd.exe", and it's informally called the
> "Command Prompt", 

Thanks for the correction, I always mix up cmd/command . exe/com. I fear
this won't be the last time either -- I wish there was a good mnemonic for
which is which.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Mon Dec  5 21:48:27 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 06 Dec 2016 13:48:27 +1100
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
 <878truq75m.fsf@elektro.pacujo.net>
 <f9ea9c78-fcdc-d909-5c4f-732865cf34cf@gmail.com>
 <mailman.64.1480982929.2253.python-list@python.org>
Message-ID: <584626fc$0$1609$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 11:08 am, Michael Torrie wrote about systemd:

> I have yet to see any evidence of this Pyonguang situation.

Let me guess... you're running a single-user Linux box?

Fortunately, I've managed to avoid needing to personally interact with
systemd at all. But over the last year or so, I've had to listen to a
continual chorus of complaints from the sys admins I work with as they
struggle to adapt our code to the Brave New World of systemd.

Let me put it this way: one of our techs took it upon himself to migrate our
Python code base from Python 2.6 to 3.4, some tens of thousands of lines.
It took him half of one afternoon.

In comparison, migrating to systemd has given us nothing but headaches, and
invariably when we try asking for help on the main systemd IRC channel
we're told that we're wrong for wanting to do what we want to do.

Not just "systemd can't do that", but "you shouldn't do that".

Why not? We used to do it, and it is necessary for our application.

"Because its wrong."




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From nathan.ernst at gmail.com  Mon Dec  5 21:51:01 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Mon, 5 Dec 2016 20:51:01 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>
 <mailman.62.1480979436.2253.python-list@python.org>
 <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPjBxg83ikcJ2-LNM1T504GgOMkwW136-YOh4n3wuY0+BMo3mg@mail.gmail.com>

Ifyou're running on Windows 10, at least, you can soon purge that memory.
command.com doesn't exist (may never have existed on Win2k, XP, Vista, 7,
8, 8.1 or 10). If I try and run either "command" or "command.com" from
Win10, both say command cannot be found.

IIRC, command.com was a relic of Win9x running on top of DOS and was a
16-bit executable, so inherently crippled (and probably never support by
the NT kernel). Whereby cmd.exe coexisted but ran in a 32-bit context.

On Mon, Dec 5, 2016 at 8:44 PM, Steve D'Aprano <steve+python at pearwood.info>
wrote:

> On Tue, 6 Dec 2016 10:09 am, eryk sun wrote:
>
> > On Mon, Dec 5, 2016 at 4:49 PM, Steve D'Aprano
> > <steve+python at pearwood.info> wrote:
> >>
> >> You've never used cmd.com or command.exe? "The DOS prompt"?
> >
> > The default Windows shell is "cmd.exe", and it's informally called the
> > "Command Prompt",
>
> Thanks for the correction, I always mix up cmd/command . exe/com. I fear
> this won't be the last time either -- I wish there was a good mnemonic for
> which is which.
>
>
>
> --
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From nathan.ernst at gmail.com  Mon Dec  5 21:56:25 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Mon, 5 Dec 2016 20:56:25 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <584626fc$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
 <878truq75m.fsf@elektro.pacujo.net>
 <f9ea9c78-fcdc-d909-5c4f-732865cf34cf@gmail.com>
 <mailman.64.1480982929.2253.python-list@python.org>
 <584626fc$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPjBxg9kAmHvOpsGzxeGZnXyP+5ny_m1zmYSi07JM+h0Fj6BOg@mail.gmail.com>

OT, but I'm curious, do they explain *why* it's wrong and give an
alternative, or just outright deride it as "the wrong way". I ask because
I've read similar complaints about the community around systemd, but as it
rarely affects me personally, I've never bothered to care.

On Mon, Dec 5, 2016 at 8:48 PM, Steve D'Aprano <steve+python at pearwood.info>
wrote:

> On Tue, 6 Dec 2016 11:08 am, Michael Torrie wrote about systemd:
>
> > I have yet to see any evidence of this Pyonguang situation.
>
> Let me guess... you're running a single-user Linux box?
>
> Fortunately, I've managed to avoid needing to personally interact with
> systemd at all. But over the last year or so, I've had to listen to a
> continual chorus of complaints from the sys admins I work with as they
> struggle to adapt our code to the Brave New World of systemd.
>
> Let me put it this way: one of our techs took it upon himself to migrate
> our
> Python code base from Python 2.6 to 3.4, some tens of thousands of lines.
> It took him half of one afternoon.
>
> In comparison, migrating to systemd has given us nothing but headaches, and
> invariably when we try asking for help on the main systemd IRC channel
> we're told that we're wrong for wanting to do what we want to do.
>
> Not just "systemd can't do that", but "you shouldn't do that".
>
> Why not? We used to do it, and it is necessary for our application.
>
> "Because its wrong."
>
>
>
>
> --
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From best_lay at yahoo.com  Mon Dec  5 22:27:10 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 21:27:10 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <9e4abd0f-933e-981b-527e-2f019460aac7@gmail.com>
 <mailman.67.1480987563.2253.python-list@python.org>
Message-ID: <Uvidnf9QjuqTrdvFnZ2dnUU7-Y2dnZ2d@giganews.com>

On Mon, 05 Dec 2016 18:25:58 -0700, Michael Torrie wrote:

> I think Python is a good choice for such a utility, but I agree it is
> much better to rely on these external utilities as children to do the
> platform-dependent work, rather than try to re-implement everything in
> Python.  A long time ago I wrote a simple wrapper to Popen that would
> run a command and return the standard out and standard error to me.

My rational is that all Linux distros are not created equal.
One comes with one certain set of utilities and another can
have different ones.  I can't always depend on a given
utility being there.  And there is not way to know the
names of same utility across all distros.  This is especially
a problem when comparing .rpm with .deb based distros.

In cases where I have to use an external program, I mean in
cases where there is no apparent Python solution, I check
for the existence of that program and just skip over that
section of the code if it is not found.

BTW here is a link for the Bash script I mentioned in case
you would like to take a look at it.  The guy who wrote
it had only been learning Bash for a few months.  Not bad.
I have tried to tweak an interest in him for Python but
he sticks with Bash.  He says that is the best language
for programming on Linux and he is not interested in GUI
programming.

https://github.com/marek-novotny/linfo

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From torriem at gmail.com  Mon Dec  5 22:41:07 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 20:41:07 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <584626fc$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
 <878truq75m.fsf@elektro.pacujo.net>
 <f9ea9c78-fcdc-d909-5c4f-732865cf34cf@gmail.com>
 <mailman.64.1480982929.2253.python-list@python.org>
 <584626fc$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <9d0aecc7-fb85-96ec-cb29-21d94cd6d889@gmail.com>

On 12/05/2016 07:48 PM, Steve D'Aprano wrote:
> On Tue, 6 Dec 2016 11:08 am, Michael Torrie wrote about systemd:
> 
>> I have yet to see any evidence of this Pyonguang situation.
> 
> Let me guess... you're running a single-user Linux box?

No I've done it on servers that weren't single-user (mail and web
servers in particular).  Not sure what you're getting at there, or why
that's relevant.  All Linux machines are set up and run as multi-user
boxes anyway.  Even my laptop that only I use.

> Fortunately, I've managed to avoid needing to personally interact with
> systemd at all. But over the last year or so, I've had to listen to a
> continual chorus of complaints from the sys admins I work with as they
> struggle to adapt our code to the Brave New World of systemd.
> 
> Let me put it this way: one of our techs took it upon himself to migrate our
> Python code base from Python 2.6 to 3.4, some tens of thousands of lines.
> It took him half of one afternoon.
>
> In comparison, migrating to systemd has given us nothing but headaches, and
> invariably when we try asking for help on the main systemd IRC channel
> we're told that we're wrong for wanting to do what we want to do.

Well since I have no idea what you were trying to do I can't comment.

> Not just "systemd can't do that", but "you shouldn't do that".
> 
> Why not? We used to do it, and it is necessary for our application.

That sounds frustrating, but of course I've heard similar stories from
folks moving to Python 3. :)

> "Because its wrong."

I've yet to encounter any of those kind of problems your admins
apparently had.  Also systemd in no way prevents you from using init
scripts or even inetd services if you choose, so there's always a
fallback position.

I'll have to specifically ask my friend who works for Bluehost about any
systemd troubles next time I speak with her.


From python.list at tim.thechases.com  Mon Dec  5 22:42:52 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Mon, 5 Dec 2016 21:42:52 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
Message-ID: <20161205214252.594f722f@bigbox.christie.dr>

On 2016-12-05 18:26, Wildman via Python-list wrote:
> On Mon, 05 Dec 2016 16:08:57 -0600, Tim Chase wrote:
> 
> > On 2016-12-05 14:58, Wildman via Python-list wrote:
> >> I there a way to detect what the Linux runlevel is from
> >> within a Python program?  I would like to be able to do
> >> it without the use of an external program such as 'who'
> >> or 'runlevel'.
> > 
> > You can use something like
> > 
> > https://gist.github.com/likexian/f9da722585036d372dca
> > 
> > to parse the /var/run/utmp contents.  Based on some source-code
> > scrounging, it looks like you want the first field to be "1" for
> > the "runlevel" account.  To extract the actual runlevel, you can
> > take the PID value from the second column ("53" in my example
> > here) and take it's integer value mod 256 (AKA "& 0xff") to get
> > the character value.  So chr(int("53") & 0xff) returns "5" in my
> > case, which is my runlevel.
> > 
> > Additional links I found helpful while searching:
> > 
> > https://casper.berkeley.edu/svn/trunk/roach/sw/busybox-1.10.1/miscutils/runlevel.c
> > https://github.com/garabik/python-utmp
> 
> That is exactly the kind of thing I was looking for.  Thank you.
> Now all I have to do is get it to work with Python3.

This works based on my poking at it in both Py2 and Py3:

  import struct
  from collections import namedtuple

  try:
    basestring
  except NameError:
    basestring = str

  UTMP = namedtuple("UTMP", [
      "ut_type", # Type of record
      "ut_pid", # PID of login process
      "ut_line", # Device name of tty - "/dev/"
      "ut_id", # Terminal name suffix, or inittab(5) ID
      "ut_user", # Username
      "ut_host", # Hostname for remote login, or kernel version for run-level messages
      "e_termination", # Process termination status
      "e_exit", # Process exit status
      "ut_session", # Session ID (getsid(2)), used for windowing
      "tv_sec", # Seconds
      "tv_usec", # Microseconds
      "ut_addr_v6a", # Internet address of remote host; IPv4 address uses just ut_addr_v6[0]
      "ut_addr_v6b", # Internet address of remote host; IPv4 address uses just ut_addr_v6[0]
      "ut_addr_v6c", # Internet address of remote host; IPv4 address uses just ut_addr_v6[0]
      "ut_addr_v6d", # Internet address of remote host; IPv4 address uses just ut_addr_v6[0]
      #"__unused", # Reserved for future use
      ])

  XTMP_STRUCT = "hi32s4s32s256shhiiiiiii20x"
  XTMP_STRUCT_SIZE = struct.calcsize(XTMP_STRUCT)

  # ut_types
  EMPTY = 0
  RUN_LVL = 1
  BOOT_TIME = 2
  OLD_TIME = 3
  NEW_TIME = 4
  INIT_PROCESS = 5 # Process spawned by "init"
  LOGIN_PROCESS = 6 # A "getty" process

  DEFAULT_UTMP = "/var/run/utmp"

  def parse_utmp(utmp_fname=DEFAULT_UTMP):
      with open(utmp_fname, "rb") as f:
          while True:
              bytes = f.read(XTMP_STRUCT_SIZE)
              if not bytes:
                  break
              bits = struct.unpack(XTMP_STRUCT, bytes)
              bits = [
                  bit.rstrip('\0') if isinstance(bit, basestring) else bit
                  for bit
                  in bits
                  ]
              yield UTMP(*bits)

  def filter(ut_type, utmp_fname=DEFAULT_UTMP):
      for utmp in parse_utmp(utmp_fname):
          if utmp.ut_type == ut_type:
              yield utmp

  def get_runlevel(utmp_fname=DEFAULT_UTMP):
      return chr(next(filter(RUN_LVL, utmp_fname)).ut_pid & 0xFF)

  if __name__ == "__main__":
      print("Runlevel: %s" % get_runlevel())


-tkc








From torriem at gmail.com  Mon Dec  5 22:46:22 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 20:46:22 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <Uvidnf9QjuqTrdvFnZ2dnUU7-Y2dnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <9e4abd0f-933e-981b-527e-2f019460aac7@gmail.com>
 <mailman.67.1480987563.2253.python-list@python.org>
 <Uvidnf9QjuqTrdvFnZ2dnUU7-Y2dnZ2d@giganews.com>
Message-ID: <0c551289-3eab-18bb-635a-615a5480133e@gmail.com>

On 12/05/2016 08:27 PM, Wildman via Python-list wrote:
> On Mon, 05 Dec 2016 18:25:58 -0700, Michael Torrie wrote:
> 
>> I think Python is a good choice for such a utility, but I agree it is
>> much better to rely on these external utilities as children to do the
>> platform-dependent work, rather than try to re-implement everything in
>> Python.  A long time ago I wrote a simple wrapper to Popen that would
>> run a command and return the standard out and standard error to me.
> 
> My rational is that all Linux distros are not created equal.
> One comes with one certain set of utilities and another can
> have different ones.  I can't always depend on a given
> utility being there.  And there is not way to know the
> names of same utility across all distros.  This is especially
> a problem when comparing .rpm with .deb based distros.

Well this is a problem regardless of which scripting language you choose
and the solutions will be the same.

> In cases where I have to use an external program, I mean in
> cases where there is no apparent Python solution, I check
> for the existence of that program and just skip over that
> section of the code if it is not found.

Sure. That's probably reasonable.

> 
> BTW here is a link for the Bash script I mentioned in case
> you would like to take a look at it.  The guy who wrote
> it had only been learning Bash for a few months.  Not bad.
> I have tried to tweak an interest in him for Python but
> he sticks with Bash.  He says that is the best language
> for programming on Linux and he is not interested in GUI
> programming.
> 
> https://github.com/marek-novotny/linfo

That all depends on what he's programming. For anything a user interacts
with, Bash is a pretty poor tool.  But Bash is really good at
shell-scripting system tasks.  It treats external commands as
first-class entities, and process control and I/O piping is integrated
into the syntax of the language.  On the other hand, Python is a good
language but it's not particularly well suited to shell scripting
(wasn't designed for that purpose), though it does have some facilities
like generators that make certain forms of system programming really
slick. In short they overlap in purpose, but they aren't good at the
same things.  However when a bash script gets too long, I'll often
switch to Python, using my run wrapper and generator filters to process
the output of external commands.

Personally for a script of this type, I'd probably stick with Bash
myself. Which by the way is what inxi is written in.


From torriem at gmail.com  Mon Dec  5 22:57:59 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 5 Dec 2016 20:57:59 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
Message-ID: <abf92eb1-c35f-09ac-d008-d3170f9737ff@gmail.com>

On 12/05/2016 07:25 PM, Dennis Lee Bieber wrote:
> On Mon, 5 Dec 2016 20:55:41 +0000, BartC <bc at freeuk.com> declaimed
> the following:
> 
>> This was a response to someone saying the wildcard param needs to
>> be at the end. There need be no such restriction if handled
>> properly (ie. no auto-expansion).
>> 
> That only applies if there is no prefix indicating a command option 
> from a file name.
> 
>> but one of the files in the list is called "-lm", or some other
>> option
> 
> -lm is not a valid file name on the OS's that use - as an option 
> prefix.

"-" is perfectly valid in a filename on Linux. Getting apps to recognize
it as a filename and not an argument is another story. Convention is to
allow an argument "--" that tells the arg parser that everything
following that is not an argument but a parameter which may or may not
be a file--BartC seems stuck on this point, but parameters could be
anything from urls to string messages to numbers. They don't have to be
files and they in fact could begin with "/" if the program allowed it.

I argue that having convention used by programs and conventions sued by
shells instead of some kind of arbitrary OS-enforced scheme is
inherently more flexible and has its own consistency (everything is
explicit from any program's point of view).

>> Without expansion, input is easy to parse: filespec, followed by 
>> optional options. But with expansion, now you have to decide if a 
>> particular argument is an option, or a filename.
>> 
> And you do that using a delimiter character that is not valid in 
> filenames. On Windows, file names can not have a /, and that is the 
> character used by the command line interpreter to indicate an option 
> follows. On UNIX, - is used as the delimiter of an option.

What is valid in a filename is not relevant here.  No one says
command-line parameters are file names if they aren't an option or
argument.  What a parameter means is up to the application (convention).
 Windows disallows / in a filename.  Linux happens to also.  But Linux
does allow '-' in a filename.  Any character can be used in a parameter
to a program except \0, from the program's point of view.

Back to shell expansion, if I had a file named "-atest.txt" and I passed
an argument to a command like this:

command -a*

The shell would in fact expand that to "-atest.txt" much to BartC's
consternation.


From best_lay at yahoo.com  Mon Dec  5 23:26:54 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 22:26:54 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <9e4abd0f-933e-981b-527e-2f019460aac7@gmail.com>
 <mailman.67.1480987563.2253.python-list@python.org>
 <Uvidnf9QjuqTrdvFnZ2dnUU7-Y2dnZ2d@giganews.com>
 <0c551289-3eab-18bb-635a-615a5480133e@gmail.com>
 <mailman.76.1480995988.2253.python-list@python.org>
Message-ID: <Uvidnf5QjuqTo9vFnZ2dnUU7-Y2dnZ2d@giganews.com>

On Mon, 05 Dec 2016 20:46:22 -0700, Michael Torrie wrote:

> On 12/05/2016 08:27 PM, Wildman via Python-list wrote:
>> On Mon, 05 Dec 2016 18:25:58 -0700, Michael Torrie wrote:
>> 
>>> I think Python is a good choice for such a utility, but I agree it is
>>> much better to rely on these external utilities as children to do the
>>> platform-dependent work, rather than try to re-implement everything in
>>> Python.  A long time ago I wrote a simple wrapper to Popen that would
>>> run a command and return the standard out and standard error to me.
>> 
>> My rational is that all Linux distros are not created equal.
>> One comes with one certain set of utilities and another can
>> have different ones.  I can't always depend on a given
>> utility being there.  And there is not way to know the
>> names of same utility across all distros.  This is especially
>> a problem when comparing .rpm with .deb based distros.
> 
> Well this is a problem regardless of which scripting language you choose
> and the solutions will be the same.

It is a problem only if you depend on the utility.

> Personally for a script of this type, I'd probably stick with Bash
> myself.

In most cases I would agree with that, but, in this case my
goal is learning Python.  I did write a couple of programs
with Bash several months ago to learn a little about it.
One will take an image and convert it into an X-Face header
and the other will take an image and convert it into a Face
header.  I later wrote GUI versions of the programs with
Python and Tkinter.

BTW, I don't depend on programming for a living.  I would be
in bad shape if I did.  It is a hobby that I greatly enjoy.
And, being in my later years, it keeps my mind sharp(er).

> Which by the way is what inxi is written in.

Yes, I was aware of that.  It's over 12,000 lines!

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Tue Dec  6 00:00:38 2016
From: best_lay at yahoo.com (Wildman)
Date: Mon, 05 Dec 2016 23:00:38 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
Message-ID: <UvidnflQjupr2NvFnZ2dnUU7-Y2dnZ2d@giganews.com>

On Mon, 05 Dec 2016 21:42:52 -0600, Tim Chase wrote:

> On 2016-12-05 18:26, Wildman via Python-list wrote:
>> On Mon, 05 Dec 2016 16:08:57 -0600, Tim Chase wrote:
>> 
>> > On 2016-12-05 14:58, Wildman via Python-list wrote:
>> >> I there a way to detect what the Linux runlevel is from
>> >> within a Python program?  I would like to be able to do
>> >> it without the use of an external program such as 'who'
>> >> or 'runlevel'.
>> > 
>> > You can use something like
>> > 
>> > https://gist.github.com/likexian/f9da722585036d372dca
>> > 
>> > to parse the /var/run/utmp contents.  Based on some source-code
>> > scrounging, it looks like you want the first field to be "1" for
>> > the "runlevel" account.  To extract the actual runlevel, you can
>> > take the PID value from the second column ("53" in my example
>> > here) and take it's integer value mod 256 (AKA "& 0xff") to get
>> > the character value.  So chr(int("53") & 0xff) returns "5" in my
>> > case, which is my runlevel.
>> > 
>> > Additional links I found helpful while searching:
>> > 
>> > https://casper.berkeley.edu/svn/trunk/roach/sw/busybox-1.10.1/miscutils/runlevel.c
>> > https://github.com/garabik/python-utmp
>> 
>> That is exactly the kind of thing I was looking for.  Thank you.
>> Now all I have to do is get it to work with Python3.
> 
> This works based on my poking at it in both Py2 and Py3:

That works perfectly.  I owe you a big thanks.  That was a
lot of work and time on your part.  I really appreciate it.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From steve+comp.lang.python at pearwood.info  Tue Dec  6 00:17:32 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 06 Dec 2016 16:17:32 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
 <abf92eb1-c35f-09ac-d008-d3170f9737ff@gmail.com>
 <mailman.77.1480996685.2253.python-list@python.org>
Message-ID: <584649ee$0$21718$c3e8da3@news.astraweb.com>

On Tuesday 06 December 2016 14:57, Michael Torrie wrote:

> "-" is perfectly valid in a filename on Linux. Getting apps to recognize
> it as a filename and not an argument is another story. Convention is to
> allow an argument "--" that tells the arg parser that everything
> following that is not an argument but a parameter which may or may not
> be a file

Another useful trick is to use a relative or absolute path to refer to a 
filename that begins with a dash:

    ./-foo

is unambiguously the file called "-foo" in the current directory, not the -f 
option.

This doesn't help when it comes to arguments which don't refer to filenames, 
hence the -- tradition.


> --BartC seems stuck on this point, but parameters could be
> anything from urls to string messages to numbers. They don't have to be
> files and they in fact could begin with "/" if the program allowed it.

Indeed.

The Unix shells are optimized for a particular use-case: system administration. 
For that, the globbing conventions etc are pretty close to optimal, but of 
course they're not the only conventions possible. Unix shells recognise this 
and allow you to escape metacharacters, and even turn off globbing altogether. 
Another alternative would be to eschew the use of a single command line and 
build your own scripting mini-language with its own conventions. That's 
especially useful if your primary use-case is to invoke your program many 
times, rather than just once.

E.g. the Postgresql interactive interpreter allows you to run multiple queries 
interactively without worrying about the shell:

https://www.postgresql.org/docs/current/static/tutorial-accessdb.html


There's no doubt that Bart has a legitimate use-case: he wants his input to be 
fed directly to his program with no pre-processing. (At least, no globbing: if 
he has given his opinion on environment variable expansion, I missed it.) Fair 
enough. Its easy to escape wildcards, but perhaps there should be an easy way 
to disable *all* pre-processing for a single command.

The fact that there is no easy, well-known way to do so indicates just how 
unusual Bart's use-case is. Linux and Unix sys admins are really good at 
scratching their own itches, and believe me, if there was widespread wish to 
disable pre-processing for a single command, after 40-odd years of Unix the 
majority of shells would support it.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From greg.ewing at canterbury.ac.nz  Tue Dec  6 01:28:52 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 06 Dec 2016 19:28:52 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24k60$ufp$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me>
Message-ID: <ean456F67hcU1@mid.individual.net>

BartC wrote:
> On 05/12/2016 19:29, Michael Torrie wrote:
> 
>> On 12/05/2016 11:50 AM, BartC wrote:
> 
> 
>>> So how do I do:
>>>
>>>     gcc *.c -lm
>>>
>>> The -lm needs to go at the end.
>>>
>>> Presumably it now needs to check each parameter seeing if it resembles a
>>> file more than it does an option? And are options automatically taken
>>> care of, or is that something that, unlike the easier wildcards, need to
>>> be processed explicitly?
> 
> 
> This was a response to someone saying the wildcard param needs to be at 
> the end. There need be no such restriction if handled properly (ie. no 
> auto-expansion).
> 
> But a similar example, suppose a syntax is:
> 
>   appl *.* [options]
> 
> but one of the files in the list is called "-lm", or some other option 
> that can do things the user didn't want (with gcc, -lm is harmless).
> 
> Without expansion, input is easy to parse: filespec, followed by 
> optional options. But with expansion, now you have to decide if a 
> particular argument is an option, or a filename.
> 
> And if there's an error in an option, you may have to abort, which means 
> throwing away that list of files which, in some cases, can run into 
> millions.
> 
>> Not having glob expansion be in individual programs makes things much
>> more explicit, from the program's point of view.  It provides an
>> interface that takes filename(s) and you provide them, either explicitly
>> (via popen with no shell), or you can do it implicitly but in an
>> interactive way via the shell using expansion.  Personal preference but
>> I believe it's a much better way because it's explicit from the
>> program's point of view as there's no question about the program's 
>> behavior.
>>
>> Different strokes for different folks as they say.
> 
> 
> I must have given ten or twenty scenarios where such auto-expansion is 
> problematical. And yet people are still in denial. It's been in Unix for 
> decades therefore there's nothing wrong with it!
> 
> But maybe people simply avoid all the situations that cause problems. 
> Interfaces are specified in a certain manner, given that input can be 
> any long stream of filenames and/or options with no order and no 
> positional significance. Non-file parameters that use ? or * are 
> prohibited. You can't do the equivalent of:
> 
>  >DIR *.b *.c
> 
> And get a list of *.b files, with a heading and summary lines, then a 
> list of *.c files with its own heading and summary. It would be one 
> monolithic list.
> 
> So everyone is working around the restrictions and the problems. Which 
> is exactly what I would have to do.
> 
> That doesn't change the fact that the Windows approach is considerably 
> more flexible and allowing more possibilities.
> 


From orgnut at yahoo.com  Tue Dec  6 02:01:32 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Mon, 5 Dec 2016 23:01:32 -0800
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.72.1480992665.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>
 <mailman.62.1480979436.2253.python-list@python.org>
 <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPjBxg83ikcJ2-LNM1T504GgOMkwW136-YOh4n3wuY0+BMo3mg@mail.gmail.com>
 <mailman.72.1480992665.2253.python-list@python.org>
Message-ID: <FOudnVCjZrrL_9vFnZ2dnUU7-VvNnZ2d@giganews.com>

On 12/05/2016 06:51 PM, Nathan Ernst wrote:
> IIRC, command.com was a relic of Win9x running on top of DOS and was a
> 16-bit executable, so inherently crippled (and probably never support by
> the NT kernel). Whereby cmd.exe coexisted but ran in a 32-bit context.

I know my 79-year-old memory is definitely subject to "senior moments" and not too reliable, but 
IIRC it was Windows prior to 9x (Win 3 and earlier) that were 16 bit and ran on top of DOS. 
Win95 was the first 32 bit version and was independent from DOS.

-- 
      -=- Larry -=-


From greg.ewing at canterbury.ac.nz  Tue Dec  6 02:08:32 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 06 Dec 2016 20:08:32 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24k60$ufp$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me>
Message-ID: <ean6fiF6nc9U1@mid.individual.net>

BartC wrote:
> But a similar example, suppose a syntax is:
> 
>   appl *.* [options]

I would be disappointed by such a syntax. What if I want
to operate on two or more files with unrelated names? With
that syntax, I can't list them explicitly in the one command.

To make that possible, the syntax would have to be

    appl filespec... [options]

i.e. allow an arbitrary number of filespecs followed by
options -- requiring the command line to be scanned looking
for options anyway.

> And if there's an error in an option, you may have to abort, which means 
> throwing away that list of files which, in some cases, can run into 
> millions.

This "millions of files" thing seems to be an imaginary
monster you've invented to try to scare people. I claim
that, to a very good approximation, it doesn't exist.
I've never encountered a directory containing a million
files, and if such a thing existed, it would be pathological
in enough other ways to make it a really bad idea.

> But maybe people simply avoid all the situations that cause problems. 
> Interfaces are specified in a certain manner, given that input can be 
> any long stream of filenames and/or options with no order and no 
> positional significance.

Exactly. The programs are designed with knowledge of the
way shells behave and are typically used.

> You can't do the equivalent of:
> 
>  >DIR *.b *.c
> 
> And get a list of *.b files, with a heading and summary lines, then a 
> list of *.c files with its own heading and summary.

Not with *that particular syntax*. You would need to
design the interface of a program to do that some other
way.

> That doesn't change the fact that the Windows approach is considerably 
> more flexible and allowing more possibilities.

At the expense of having shells with less powerful
facilities, and more inconsistent behaviour between
different programs.

One isn't better than the other. There are tradeoffs.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Dec  6 02:12:41 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 06 Dec 2016 20:12:41 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.70.1480991411.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <958c4ch1cddjp9tptf91340as9e5qmqmk1@4ax.com>
 <mailman.70.1480991411.2253.python-list@python.org>
Message-ID: <ean6nbF6p2fU1@mid.individual.net>

Dennis Lee Bieber wrote:
> -lm is not a valid file name on the OS's that use - as an option
> prefix.

It's not invalid -- you can create a file called -lm
on a unix system if you want, you just have to be a bit
sneaky about how you refer to it:

% echo foo > ./-lm
% ls
-lm
% cat ./-lm
foo

Sane people normally refrain from using such file names,
however, because of the hassle of dealing with them.

-- 
Greg


From orgnut at yahoo.com  Tue Dec  6 02:37:13 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Mon, 5 Dec 2016 23:37:13 -0800
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o24cr9$1ri$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
Message-ID: <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>

On 12/05/2016 10:50 AM, BartC wrote:

>>     And just what ARE A, C, and D?
>
> It doesn't matter, and is not the concern of the shell. It should restrict itself to the basic
> parsing that may be necessary when parameters are separated by white-space and commas, if a
> parameter can contain white-space or commas. That usually involves surrounding the parameter
> with quotes.
>
> One would be very annoyed if, reading a CSV file, where each of N values on a line correspond to
> a field of record, if one entry of "?LBC" expanded itself to a dozen entries, screwing
> everything up.
>
Now you're suggesting the _shell_ is going to read and process a CVS file???

> Shell command line processing shouldn't be attempting anything more than that.
>
I get awfully tired of your constant pontificating about your opinions.  I know they're _VERY_ 
strongly held beliefs on your part, but...  they are ONE person's opinions and are no more than 
opinions and they ain't gonna change nothin', no how, no way, not ever.

[Sorry, I'm in a bad mood today and just had to let off a little steam...]

-- 
      -=- Larry -=-


From marko at pacujo.net  Tue Dec  6 02:47:26 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 06 Dec 2016 09:47:26 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <o24ml9$74d$1@dont-email.me>
 <CAPTjJmqLd9A78T6Bx=pb=fwecnBvDTBXcHos56YCha666HQwfA@mail.gmail.com>
 <mailman.57.1480976687.2253.python-list@python.org>
 <87d1h6qaah.fsf@elektro.pacujo.net>
 <17c422f2-0e4b-5eac-3e7c-db3759091946@gmail.com>
 <mailman.61.1480977899.2253.python-list@python.org>
 <878truq75m.fsf@elektro.pacujo.net>
 <f9ea9c78-fcdc-d909-5c4f-732865cf34cf@gmail.com>
 <mailman.64.1480982929.2253.python-list@python.org>
Message-ID: <87wpfdpkhd.fsf@elektro.pacujo.net>

Michael Torrie <torriem at gmail.com>:

> On 12/05/2016 04:37 PM, Marko Rauhamaa wrote:
>> Unfortunately, I am not wholly impressed by the end result. Mogadishu
>> has been replaced by Pyongyang. Some age-old Unix principles have been
>> abandoned without clear justification. For example, I was appalled to
>> find out that a systemd unit can be configured to react on the exit
>> status of a child process of a daemon.
>
> I have yet to see any evidence of this Pyonguang situation.
>
> What is wrong with reacting to the exit status of a daemon's child
> process?

A basic black-box principle is violated. It is surprising, at least.

If I launch a child process, wait it out and ignore its exit status
code, I would think the exit status is meaningless. Not so, because the
Eye is watching.

>> Also, now D-Bus is a fixture for every daemon writer.
>
> But not directly. It need not be a dependency on the part of the
> daemon writer. In fact the daemon writer may now leave out deamonizing
> code entirely if he or she wishes. The systemd api is entirely
> optional for the daemon to use.

Systemd comes with dozens of legacy modes, and it is difficult to learn
what is systemd's "native" daemon type. However, it is evident that
"Type=notify" is it, meaning the daemon communicates with systemd
explicitly (<URL:
https://www.freedesktop.org/software/systemd/man/sd_notify.html>).

That is not necessarily a bad idea. Daemons have traditionally been
lousy at communicating their statuses appropriately. It's just that this
major requirement should be declared openly.

>> The .ini file format was a lousy choice. Why not choose JSON in this
>> day and age?
>
> But why json?

It comes with simple, rigorous, universal syntax. The .ini files don't.
That's why the unit files have brittle ad-hoc syntax.


Marko


From p.f.moore at gmail.com  Tue Dec  6 05:51:48 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 6 Dec 2016 02:51:48 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.45.1480962105.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o24a0n$m62$1@dont-email.me>
 <CAPTjJmqHOW6g3_Jg7KDgGgXKXRwSH7Yhm808dR9dL2yueZrkmw@mail.gmail.com>
 <mailman.45.1480962105.2253.python-list@python.org>
Message-ID: <f44e53df-a499-46c2-a2c0-a563f3bf8663@googlegroups.com>

On Monday, 5 December 2016 18:21:57 UTC, Chris Angelico  wrote:
> On Tue, Dec 6, 2016 at 5:02 AM, BartC <bc at freeuk.com> wrote:
> >
> > how do you tell whether the last file in an argument list is the optional
> > 'file', or the last file of the expansion of 'filespec'?
> 
> Why should you care? I have used shell globbing to pass precisely two
> parameters to a program. More often, I use this syntax, which Windows
> simply doesn't support:

You might care. I occasionally teach Unix to beginners, and a common gotcha is the fact that

    cp a/* .

copies everything from a to the current directory. But if you miss typing the ".", the behaviour is *very* different. And the cp command has no way to know or warn you that you might have mistyped.

The find command is another, more "advanced" example.

    find . -name foo*

works as "expected" as long as there's no file that matches the glob in the current directory (because in that case the shell passes the pattern through unchanged). But users who get used to this behaviour get a nasty surprise when they hit a case where it doesn't apply and they *need* to quote.

It's a trade-off. Unix makes shells do globbing, so programs don't have to, but as a consequence they have no means of seeing whether globbing occurred, or switching it off for particular argument positions. Windows chooses to make the trade-off a different way.

Paul


From p.f.moore at gmail.com  Tue Dec  6 05:57:38 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 6 Dec 2016 02:57:38 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.41.1480958057.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CANc-5UwWJ7T7k43=cgPsyQuQh7qn4LbCYa=8Hqr2BVdRgd4PMg@mail.gmail.com>
 <mailman.41.1480958057.2253.python-list@python.org>
Message-ID: <0c7bb263-d2b0-4c1f-b9cd-d741610eaa4d@googlegroups.com>

On Monday, 5 December 2016 17:14:27 UTC, Skip Montanaro  wrote:
> ISTR that the way DOS/Windows operate at the text prompt level was
> modeled on VMS. As you indicated, each command was responsible for its
> own "globbing". I've never programmed in DOS or Windows, and its been
> decades since I programmed in VMS, but I imagine that both
> environments probably provide some standard sort of globbing library.

Technically, the primitive "launch an executable" operation in Windows takes a *command line* to pass to the new process, rather than a list of arguments. The argv convention comes to Windows via C, which is derived from Unix.

So the C runtime historically provided argv splitting to match C semantics, and added globbing as a convenience "because people were used to it from Unix" (even though in Unix it was a shell feature, not a C/application feature). There's also an OS API to do cmdline->argv conversion, for programs that don't want to rely on the C runtime capability.

The result is the same, though - in Windows, applications (or the language runtime) handle globbing, but in Unix the shell does.

Paul


From bc at freeuk.com  Tue Dec  6 06:21:20 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 11:21:20 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
Message-ID: <o266t3$bnd$1@dont-email.me>

On 06/12/2016 07:37, Larry Hudson wrote:

> Now you're suggesting the _shell_ is going to read and process a CVS
> file???

What follows a shell command is a set of values a,b,c,d,e. What is 
encountered in a CSV is a set of values a,b,c,d,e. You really can't see 
the similarity?

Suppose instead of:

   command a b c d e

The interface was changed to be more interactive:

   command
   Input:  a b c d e

So parameters are not entered on the command line, but are prompted for. 
Do you think entering a b c d e here should give exactly the same 
results as doing so on the command line?

As far as any user is concerned, they should. But they can't because in 
the in-line example, parameters could be expanded. And there seems to be 
no way of turning that off, without changing the input (eg. quotes 
around parameters).

>> Shell command line processing shouldn't be attempting anything more
>> than that.
>>
> I get awfully tired of your constant pontificating about your opinions.

Opinions based on facts: I've given a dozen examples where the shell's 
auto-expansion can screw things up. And I can easily come up with more, 
as have others in the thread.

People's attitudes seem to be 'So don't that'. Or, 'So what?'.

Which suggests there is an actual problem that is being brushed under 
the carpet.

> I know they're _VERY_ strongly held beliefs on your part, but...  they
> are ONE person's opinions and are no more than opinions and they ain't
> gonna change nothin', no how, no way, not ever.

No they're not. But the auto-expansion of parameters by default is still 
wrong.

> [Sorry, I'm in a bad mood today and just had to let off a little steam...]

People spout off about Windows ALL the time. 'So what?'

-- 
Bartc




From rosuav at gmail.com  Tue Dec  6 06:42:25 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 22:42:25 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o266t3$bnd$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me>
Message-ID: <CAPTjJmoWCuy3hADjQ_FjzUG8QQN4WL3uHEGNTP2XbnXAGHAQfA@mail.gmail.com>

On Tue, Dec 6, 2016 at 10:21 PM, BartC <bc at freeuk.com> wrote:
> What follows a shell command is a set of values a,b,c,d,e. What is
> encountered in a CSV is a set of values a,b,c,d,e. You really can't see the
> similarity?
>
> Suppose instead of:
>
>   command a b c d e
>
> The interface was changed to be more interactive:
>
>   command
>   Input:  a b c d e
>
> So parameters are not entered on the command line, but are prompted for. Do
> you think entering a b c d e here should give exactly the same results as
> doing so on the command line?
>
> As far as any user is concerned, they should.

Here are two Python statements:

x = 1, 2, 3, 4, 5
f(x)

f(1, 2, 3, 4, 5)

As far as any user is concerned, these should do the exact same thing.
After all, they both have five numbers separated by commas. There's
absolutely no reason for them to do anything different. And it should
be exactly the same if you write it like this:

f(input())

and type "1, 2, 3, 4, 5" at the prompt. Right?

Why do you continue to claim that shells should do no parsing, yet
expect parsing to happen elsewhere? Why are shells somehow different?

ChrisA


From bc at freeuk.com  Tue Dec  6 06:43:54 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 11:43:54 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.69.1480990884.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
Message-ID: <o2687e$ft7$1@dont-email.me>

On 06/12/2016 02:21, Dennis Lee Bieber wrote:
> On Mon, 5 Dec 2016 18:50:30 +0000, BartC <bc at freeuk.com> declaimed the
> following:
>
>> It doesn't matter, and is not the concern of the shell. It should
>> restrict itself to the basic parsing that may be necessary when
>
> 	Another name for "shell" is "command line interpreter" -- emphasis on
> "interpreter"...
>
> 	They are languages in their own right, with their own rules.

I distinguish between proper languages and those that just process 
commands like those you type in to applications that use command line input:

   > kill dwarf with axe

You don't really expect input like:

   >HOW ARE YOU?
   VERY WELL THANKS!

[An actual demo of a board computer in the 1970s] to be transformed into:

   HOW ARE YOUA YOUB YOUC

> 	The Windows command prompt being one of the weakest -- it doesn't
> support arithmetic and local variables, nor (to my knowledge) looping
> constructs.

It does some of that, but very crudely. If you want scripting, then use 
a scripting language. There are plenty about. A typical Windows user who 
uses the command prompt will have no idea they are doing coding. And 
they're not. /I/ don't expect that on Unix either.

>> One would be very annoyed if, reading a CSV file, where each of N values
>> on a line correspond to a field of record, if one entry of "?LBC"
>> expanded itself to a dozen entries, screwing everything up.
>
> 	Meaningless argument... You are, here, /reading a data file/, not
> interpreting the contents as command lines.

They're both data.

> 	Read the Python documentation for argparse

I just tried it, but it was too complex for me to set it up so as to 
discover with it did with * arguments.

> 	Again, start with argparse... Any command line argument that is left
> after it has parsed the line can likely be considered a "filename".

Only at the end?

  And to
> handle the difference between Windows and UNIX you'd likely need something
> like:
>
> for aParm in remainingArguments:
> 	for aFile in glob.glob(aParm):
> 		do something with the file

Suppose any argument contains * or ?, isn't a filename, but happens to 
match some files in the current directory. AFAICS it will still screw up.

-- 
Bartc


From bc at freeuk.com  Tue Dec  6 06:56:26 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 11:56:26 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <ean6fiF6nc9U1@mid.individual.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <ean6fiF6nc9U1@mid.individual.net>
Message-ID: <o268uu$ig8$1@dont-email.me>

On 06/12/2016 07:08, Gregory Ewing wrote:
> BartC wrote:

>> And if there's an error in an option, you may have to abort, which
>> means throwing away that list of files which, in some cases, can run
>> into millions.
>
> This "millions of files" thing seems to be an imaginary
> monster you've invented to try to scare people. I claim
> that, to a very good approximation, it doesn't exist.
> I've never encountered a directory containing a million
> files, and if such a thing existed, it would be pathological
> in enough other ways to make it a really bad idea.

Many of my examples are based on actual experience.

One of my machines /did/ have 3.4 million files in one directory.

(The result of Firefox file caching having run amok for the best part of 
a year. Once discovered, it took 15 hours to delete them all.)

In that directory (which was on Windows but accessible via a virtual 
Linux), typing any Linux command followed by * would have required all 
3.4 million directory entries to be accessed in order to build a 3.4 
million-element argv list. I've no idea how long that would have taken.

>>  >DIR *.b *.c

> Not with *that particular syntax*. You would need to
> design the interface of a program to do that some other
> way.

EXACTLY. It's restrictive.

-- 
Bartc




From python.list at tim.thechases.com  Tue Dec  6 06:59:45 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Tue, 6 Dec 2016 05:59:45 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <UvidnflQjupr2NvFnZ2dnUU7-Y2dnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <UvidnflQjupr2NvFnZ2dnUU7-Y2dnZ2d@giganews.com>
Message-ID: <20161206055945.4e696f15@bigbox.christie.dr>

On 2016-12-05 23:00, Wildman via Python-list wrote:
> On Mon, 05 Dec 2016 21:42:52 -0600, Tim Chase wrote:
> > This works based on my poking at it in both Py2 and Py3:
> 
> That works perfectly.  I owe you a big thanks.  That was a
> lot of work and time on your part.  I really appreciate it.

It was pretty straightforward to map it into a namedtuple using the
links I provided.  The only killer for me was that the struct module
doesn't return an entry for "padding" bytes.  Which makes sense, but
threw me off for a good while as I tried to figure why my
tuple-creation was complaining about a missing parameter (fixed by
commenting out the "__unused" member of the namedtuple; could also
have kept it while changing the "20x" to "20c" in the struct
format-string)

The rest was just basic text manipulation in vim to convert the C
structs into Python code.

And despite what Bernd Nawothnig wrote:

> It looks for me like a worthless learning experience.

it gave me the opportunity to learn about the internals of the utmp
format, something that has long been a curiosity ("it's a binary log,
not text, I wonder what all is in there? But I don't have real cause
to go poking around in there to learn the answer right now.") and this
gave me the push I needed to explore that.

Glad it helped.

-tkc






From rosuav at gmail.com  Tue Dec  6 07:26:49 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 6 Dec 2016 23:26:49 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o268uu$ig8$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me>
 <ean6fiF6nc9U1@mid.individual.net> <o268uu$ig8$1@dont-email.me>
Message-ID: <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>

On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
> In that directory (which was on Windows but accessible via a virtual Linux),
> typing any Linux command followed by * would have required all 3.4 million
> directory entries to be accessed in order to build a 3.4 million-element
> argv list. I've no idea how long that would have taken.

I just asked Python to build me a 4-million-element list, and it took
no visible time - a small fraction of a second. Don't be afraid of
large argument lists. We're not writing 8088 Assembly Language
programs in 64KB of working memory here.

ChrisA


From eryksun at gmail.com  Tue Dec  6 07:39:30 2016
From: eryksun at gmail.com (eryk sun)
Date: Tue, 6 Dec 2016 12:39:30 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPjBxg83ikcJ2-LNM1T504GgOMkwW136-YOh4n3wuY0+BMo3mg@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>
 <mailman.62.1480979436.2253.python-list@python.org>
 <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPjBxg83ikcJ2-LNM1T504GgOMkwW136-YOh4n3wuY0+BMo3mg@mail.gmail.com>
Message-ID: <CACL+1avQKFbNDLx3FN7sO2L3r7yL0wWRfA=dUkHyO-n=Wnb6+A@mail.gmail.com>

On Tue, Dec 6, 2016 at 2:51 AM, Nathan Ernst <nathan.ernst at gmail.com> wrote:
> On Mon, Dec 5, 2016 at 8:44 PM, Steve D'Aprano <steve+python at pearwood.info>
> wrote:
>> On Tue, 6 Dec 2016 10:09 am, eryk sun wrote:
>>
>> > The default Windows shell is "cmd.exe", and it's informally called the
>> > "Command Prompt",
>>
>> Thanks for the correction, I always mix up cmd/command . exe/com. I fear
>> this won't be the last time either -- I wish there was a good mnemonic for
>> which is which.

There are a few executables that end in .com: chcp.com, format.com,
mode.com, more.com, and tree.com. These are 32-bit / 64-bit PE/COFF
binaries, the same as any other Windows executable.

> Ify ou're running on Windows 10, at least, you can soon purge that memory.
> command.com doesn't exist (may never have existed on Win2k, XP, Vista, 7,
> 8, 8.1 or 10). If I try and run either "command" or "command.com" from
> Win10, both say command cannot be found.

Only 32-bit versions of Windows include the NT Virtual DOS Machine
(ntvdm.exe) for running 16-bit DOS programs. Such programs expect a
8086 real-mode execution environment, in which the DOS kernel hooks
interrupt 0x21 for its system-call interface. To provide this
environment, NTVDM uses a virtual 8086 mode monitor that's built into
the 32-bit kernel.

x86-64 long mode doesn't allow switching the CPU to v86 mode, so NTVDM
isn't available in 64-bit Windows. In this case the kernel's entry
point for VDM control is hard coded to return STATUS_NOT_IMPLEMENTED
(0xC0000002), as the following disassembly shows:

    lkd> u nt!NtVdmControl
    nt!NtVdmControl:
    fffff801`ffff4710 b8020000c0      mov     eax,0C0000002h
    fffff801`ffff4715 c3              ret

> IIRC, command.com was a relic of Win9x running on top of DOS and was a
> 16-bit executable, so inherently crippled (and probably never support by
> the NT kernel).

COMMAND.COM is a 16-bit DOS program, which was the "DOS prompt" in
Windows 3.x and 9x. The versions of Windows that ran on DOS had a
complicated design (in 386 Enhanced Mode) that used a virtual 8086
monitor that ran in 32-bit protected mode. As far back as Windows 3.1,
Microsoft started replacing some DOS system services with
functionality implemented in 32-bit VxDs. Some among us may recall the
big performance improvement that 32-bit disk access provided in
Windows for Workgroups 3.11.

In Windows 9x most DOS system calls were implemented in 32-bit
protected mode VxDs; they even reflected calls in v86 mode up to the
32-bit implementation. Thus much of the implementation underlying the
Win32 API used 32-bit code in protected mode. That said, initially in
Windows 95 there were still a lot of Win32 API calls that ended up
executing real-mode 16-bit DOS calls in the system VM. The book
"Unauthorized Windows 95" analyzes this in detail.

> Whereby cmd.exe coexisted but ran in a 32-bit context.

cmd.exe (command prompt) is a Windows application -- for the most
part, though it does go beyond the Windows API to peek at the NT
process environment block (PEB) of child processes. It was ported to
but never distributed with Windows 9x. On Windows 9x you instead had
an actual DOS prompt that ran COMMAND.COM in virtual 8086 mode.


From greg.ewing at canterbury.ac.nz  Tue Dec  6 07:40:04 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Wed, 07 Dec 2016 01:40:04 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o266t3$bnd$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me>
Message-ID: <eanpt6FbbkcU1@mid.individual.net>

BartC wrote:
> I've given a dozen examples where the shell's 
> auto-expansion can screw things up.

Only because you're taking Windows conventions and trying
to apply them to Unix.

That's like somebody from the USA visiting Britain and
thinking "OMG! These people are all going to get themselves
killed, they're driving on the wrong side of the road!"

-- 
Greg


From jon+usenet at unequivocal.eu  Tue Dec  6 08:05:39 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Tue, 6 Dec 2016 13:05:39 -0000 (UTC)
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <ean6fiF6nc9U1@mid.individual.net>
 <o268uu$ig8$1@dont-email.me>
 <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>
 <mailman.80.1481027212.2253.python-list@python.org>
Message-ID: <slrno4ddvj.298.jon+usenet@sable.unequivocal.eu>

On 2016-12-06, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
>> In that directory (which was on Windows but accessible via a virtual Linux),
>> typing any Linux command followed by * would have required all 3.4 million
>> directory entries to be accessed in order to build a 3.4 million-element
>> argv list. I've no idea how long that would have taken.
>
> I just asked Python to build me a 4-million-element list, and it took
> no visible time - a small fraction of a second. Don't be afraid of
> large argument lists. We're not writing 8088 Assembly Language
> programs in 64KB of working memory here.

To be fair, literally just now I couldn't run the command I wanted to:

  sed -i -e 's/foo/bar/g' /dir/ectory/*.txt

because there were 300,000 files matching the glob, and 'sed' can't
cope with that many command-line arguments. I had to do this instead:

  find /dir/ectory -name '*.txt' -exec sed -i -e 's/foo/bar/g' {} \;

(Yes, I could also have done something with 'xargs' instead, which
would've been slower to write and quicker to run.)

However please don't take that to mean I agree with BartC - he's
clearly just reacting with instinctive hostility to the unknown.


From eryksun at gmail.com  Tue Dec  6 08:08:31 2016
From: eryksun at gmail.com (eryk sun)
Date: Tue, 6 Dec 2016 13:08:31 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me>
 <ean6fiF6nc9U1@mid.individual.net> <o268uu$ig8$1@dont-email.me>
 <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>
Message-ID: <CACL+1avD311T8f8Pgb6LkQhYZjO5cCoBmNG4UXu2+qEMXPg9Tg@mail.gmail.com>

On Tue, Dec 6, 2016 at 12:26 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
>> In that directory (which was on Windows but accessible via a virtual Linux),
>> typing any Linux command followed by * would have required all 3.4 million
>> directory entries to be accessed in order to build a 3.4 million-element
>> argv list. I've no idea how long that would have taken.
>
> I just asked Python to build me a 4-million-element list, and it took
> no visible time - a small fraction of a second. Don't be afraid of
> large argument lists. We're not writing 8088 Assembly Language
> programs in 64KB of working memory here.

The problem isn't building an arbitrary list with millions of
elements. The problem is the time it would take to read millions of
filenames from a directory. It depends on the performance of the disk
and filesystem. Be careful with globbing. Think about the consequences
before running a command, especially if you're in the habit of
creating directories with hundreds of thousands, or millions, of
files. It's not a problem that I've ever had to deal with.


From bc at freeuk.com  Tue Dec  6 08:25:21 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 13:25:21 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <eanpt6FbbkcU1@mid.individual.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
Message-ID: <o26e5l$4k6$1@dont-email.me>

On 06/12/2016 12:40, Gregory Ewing wrote:
> BartC wrote:
>> I've given a dozen examples where the shell's auto-expansion can screw
>> things up.
>
> Only because you're taking Windows conventions and trying
> to apply them to Unix.

What, the convention of NOT assuming that any command parameter that 
uses * or ? MUST refer to whatever set of filenames happen to match in 
the current directory? And to then insert N arbitrary filenames in the 
parameter list.

That's a pretty good convention, yes?!

(Or should that be yesx! yesy!)

> That's like somebody from the USA visiting Britain and
> thinking "OMG! These people are all going to get themselves
> killed, they're driving on the wrong side of the road!"

Or someone from Britain visiting the USA and saying OMG, everyone's got 
a gun! Suppose someone runs amok and shoots everybody!

-- 
Bartc


From bc at freeuk.com  Tue Dec  6 08:28:44 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 13:28:44 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.80.1481027212.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <ean6fiF6nc9U1@mid.individual.net>
 <o268uu$ig8$1@dont-email.me>
 <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>
 <mailman.80.1481027212.2253.python-list@python.org>
Message-ID: <o26ebv$4k6$2@dont-email.me>

On 06/12/2016 12:26, Chris Angelico wrote:
> On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
>> In that directory (which was on Windows but accessible via a virtual Linux),
>> typing any Linux command followed by * would have required all 3.4 million
>> directory entries to be accessed in order to build a 3.4 million-element
>> argv list. I've no idea how long that would have taken.
>
> I just asked Python to build me a 4-million-element list, and it took
> no visible time - a small fraction of a second. Don't be afraid of
> large argument lists. We're not writing 8088 Assembly Language
> programs in 64KB of working memory here.

Haven't people been saying that Unix has been doing this for 40 years?

Some systems /did/ have little memory, and while there won't have been 
the capacity for that many files, some file devices were slow.

-- 
Bartc



From rosuav at gmail.com  Tue Dec  6 08:34:34 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 7 Dec 2016 00:34:34 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26e5l$4k6$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
Message-ID: <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>

On Wed, Dec 7, 2016 at 12:25 AM, BartC <bc at freeuk.com> wrote:
> What, the convention of NOT assuming that any command parameter that uses *
> or ? MUST refer to whatever set of filenames happen to match in the current
> directory? And to then insert N arbitrary filenames in the parameter list.
>
> That's a pretty good convention, yes?!

Right! And while you're at it, stop assuming that percent signs have
meaning, that quotes have meaning, etc, etc, etc. Right? And why
should the enter key be significant - what if you want to send more
than one line of command line arguments?

ChrisA


From python.list at tim.thechases.com  Tue Dec  6 08:51:48 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Tue, 6 Dec 2016 07:51:48 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
Message-ID: <20161206075148.45f3310b@bigbox.christie.dr>

On 2016-12-06 01:14, Bernd Nawothnig wrote:
> > I am a hobby programmer and I've been trying to learn python
> > for a few months now.  The program is 'worthlessware' but it
> > is a 'learning experience' for me.    
> 
> It looks for me like a worthless learning experience.

Eh, one person's "worthless learning experience" is another person's
curiosity-satisfying education.  I found it an exercise in learning
something new about the utmp log format (previously an opaque binary
blob)

> > A friend wrote a similar program in Bash script and I have been
> > working on translating it to Python.  
> 
> Stay with shell script for such tasks. It is never a good idea to
> choose the programming language before closer evaluating the
> problem.

Based on the OP's description, this is a small part of a much larger
program.  And I would personally rather maintain a large Python
code-base than a large Bash code-base.

-tkc






From bc at freeuk.com  Tue Dec  6 08:52:20 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 13:52:20 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.83.1481031282.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
Message-ID: <o26fo7$a7s$1@dont-email.me>

On 06/12/2016 13:34, Chris Angelico wrote:
> On Wed, Dec 7, 2016 at 12:25 AM, BartC <bc at freeuk.com> wrote:
>> What, the convention of NOT assuming that any command parameter that uses *
>> or ? MUST refer to whatever set of filenames happen to match in the current
>> directory? And to then insert N arbitrary filenames in the parameter list.
>>
>> That's a pretty good convention, yes?!
>
> Right! And while you're at it, stop assuming that percent signs have
> meaning, that quotes have meaning, etc, etc, etc. Right? And why
> should the enter key be significant - what if you want to send more
> than one line of command line arguments?

But those would be silly.

Some special syntax is known about: | < and > for example. % less so 
(I've never, ever used it in live input AFAIK).

This auto-expansion causes so many problems, places so many restrictions 
on what can be done, that it doesn't make sense. Auto-expanding 
parameters is such a big deal, that it should not be the default 
behaviour; it needs something to tell the command processor to expand.

Then you don't get utterly ridiculous and dangerous behaviour such as 
the cp example Paul Moore came up with (that trumps most of mine actually):

Start with a directory containing two files c.c and d.c. You want to 
copy all .c files elsewhere, but accidentally type this:

    > cp *.c

which ends up doing:

    > cp c.c d.c

cp (or a program wanting to do anything like this) expects two arguments 
to be entered. But with auto-expansion, that is impossible to determine.

And the justification? Well, %ENVIRONMENTVARIABLE% gets converted in 
Windows, so why not?!

-- 
Bartc



From dr.roman.graf at gmail.com  Tue Dec  6 09:21:54 2016
From: dr.roman.graf at gmail.com (dr.roman.graf at gmail.com)
Date: Tue, 6 Dec 2016 06:21:54 -0800 (PST)
Subject: Django broken pipe error
Message-ID: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>

Hi,

I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that apparently is a not fixed Django bug. Does anybody now how to fix it? I've been searching a lot and didn't find any solution.

This error happens very irregularly by Post request in Django. Sometimes it works sometimes not: 

[06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 129, in __init__
    super(WSGIRequestHandler, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 38224)

It is also described in: https://www.reddit.com/r/joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/

On https://bugs.python.org/issue14574 is stated that this error should already be fixed but apparently not.

Best regards,

Roman


From p.f.moore at gmail.com  Tue Dec  6 09:25:39 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 6 Dec 2016 06:25:39 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26e5l$4k6$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
Message-ID: <9c31ab27-5535-4ec2-8e6f-05201c057bd0@googlegroups.com>

On Tuesday, 6 December 2016 13:25:40 UTC, BartC  wrote:
> On 06/12/2016 12:40, Gregory Ewing wrote:
> > BartC wrote:
> >> I've given a dozen examples where the shell's auto-expansion can screw
> >> things up.
> >
> > Only because you're taking Windows conventions and trying
> > to apply them to Unix.
> 
> What, the convention of NOT assuming that any command parameter that 
> uses * or ? MUST refer to whatever set of filenames happen to match in 
> the current directory? And to then insert N arbitrary filenames in the 
> parameter list.

Correct - with the exception that it's not that it MUST - there's ways to prevent that happening, just ones that aren't the default.

> That's a pretty good convention, yes?!

Yes. It has its benefits.

> (Or should that be yesx! yesy!)

No, because the rules for text in an email are different from those for text in a shell. But you seem to be arguing that the rules should be the same everywhere, so maybe in your world, yes it should be. Most other people understand the concept of context, though.

Paul


From random832 at fastmail.com  Tue Dec  6 09:49:46 2016
From: random832 at fastmail.com (Random832)
Date: Tue, 06 Dec 2016 09:49:46 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
Message-ID: <1481035786.3402203.810123945.70F80B06@webmail.messagingengine.com>

On Mon, Dec 5, 2016, at 21:21, Dennis Lee Bieber wrote:
> 	They are languages in their own right, with their own rules.
> 
> 	The Windows command prompt being one of the weakest -- it doesn't
> support arithmetic and local variables, nor (to my knowledge) looping
> constructs. BAT files are limited to something like 9 parameters (which
> may
> be the only argument for not expanding wildcards at the command line
> level).

There are only nine that you can name explicitly, but there's no
obstacle to handling more with shift or %*. Also, there is a 'for' loop,
though the syntax is somewhat arcane (and you can always loop with
if/goto)

It can do arithmetic with 'set /a', and there is a 'setlocal' command
for local variable scopes.


From random832 at fastmail.com  Tue Dec  6 10:05:57 2016
From: random832 at fastmail.com (Random832)
Date: Tue, 06 Dec 2016 10:05:57 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <FOudnVCjZrrL_9vFnZ2dnUU7-VvNnZ2d@giganews.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <58459a81$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1au==3BazJ3YTu_5K9m5aswekDqn5Deu0inDCH-DaOXfEw@mail.gmail.com>
 <mailman.62.1480979436.2253.python-list@python.org>
 <584625fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPjBxg83ikcJ2-LNM1T504GgOMkwW136-YOh4n3wuY0+BMo3mg@mail.gmail.com>
 <mailman.72.1480992665.2253.python-list@python.org>
 <FOudnVCjZrrL_9vFnZ2dnUU7-VvNnZ2d@giganews.com>
Message-ID: <1481036757.3406963.810142769.280D13BE@webmail.messagingengine.com>

On Tue, Dec 6, 2016, at 02:01, Larry Hudson via Python-list wrote:
> On 12/05/2016 06:51 PM, Nathan Ernst wrote:
> > IIRC, command.com was a relic of Win9x running on top of DOS and was a
> > 16-bit executable, so inherently crippled (and probably never support by
> > the NT kernel). Whereby cmd.exe coexisted but ran in a 32-bit context.
> 
> I know my 79-year-old memory is definitely subject to "senior moments"
> and not too reliable, but 
> IIRC it was Windows prior to 9x (Win 3 and earlier) that were 16 bit and
> ran on top of DOS. 
> Win95 was the first 32 bit version and was independent from DOS.

Yes but there was no* 32-bit windows command interpreter - it ran DOS in
a virtual machine inside it. Windows 3 did the same, actually - the real
architecture of Windows/386 was a 32-bit protected mode host kernel
called VMM32.VXD that ran all of Windows in one virtual machine and each
DOS window in another one, leading to the odd consequence of there being
cooperative multitasking between Windows apps but pre-emptive
multitasking between DOS apps [and between them and Windows].

The fact that Windows was launched at boot by running "win.com" (either
in autoexec.bat or manually at the command line) created a *perception*
that windows ran "on top of DOS", but running it really *replaced* DOS
in memory, putting the CPU into protected mode and everything. The
ability to boot into (or exit to) DOS was because people still did real
work (and games) in DOS and the virtual environment of DOS-on-Windows
didn't perform well enough to be sufficient.

*Well, I vaguely remember a version of cmd.exe that would run on Windows
98 floating around back in the day, but it certainly didn't come with
the OS. It might have been a pirated Windows NT component.


From torriem at gmail.com  Tue Dec  6 10:44:57 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 08:44:57 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2687e$ft7$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
Message-ID: <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>

On 12/06/2016 04:43 AM, BartC wrote:
>> 	Read the Python documentation for argparse
> 
> I just tried it, but it was too complex for me to set it up so as to 
> discover with it did with * arguments.
> 
>> 	Again, start with argparse... Any command line argument that is left
>> after it has parsed the line can likely be considered a "filename".
> 
> Only at the end?

No, that's not what he said. After arguments have been parsed out and
dealt with, whatever is left can be retrieved as the parameters (whether
those are filenames or urls or something.  All remaining parameters.
Wherever they appeared.  Some argument parsers do require all arguments
to be first on the command line.  argparse is not one of them.  BSD
tools typically do want args first.  And actually a lot of windows
applications are extremely picky about where the arguments come vs the
"filespec" parameters.

>   And to
>> handle the difference between Windows and UNIX you'd likely need something
>> like:
>>
>> for aParm in remainingArguments:
>> 	for aFile in glob.glob(aParm):
>> 		do something with the file
> 
> Suppose any argument contains * or ?, isn't a filename, but happens to 
> match some files in the current directory. AFAICS it will still screw up.

Precisely! And you can bet there is probably more than one Windows
program out there that incorrectly makes this assumption and does the
wrong thing.  Or the opposite is true and there are programs that expect
no globs and can do nothing with them.  And for such a buggy program
there's not a darn thing the user can do to escape the glob or otherwise
tell the program it's not a glob.

That's why I would far rather place globbing in control of the shell
where a user can properly deal with it, escape it, or otherwise disable
it when necessary.

Yes shell expansion has it's gotchas.  But those can all be learned,
whereas it's much harder to learn and remember all the gotchas and bugs
of many individual applications' unique ways of dealing with globs. I'd
rather deal with shells.



From best_lay at yahoo.com  Tue Dec  6 11:18:22 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 06 Dec 2016 10:18:22 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
Message-ID: <GbmdnUBEet5TedvFnZ2dnUU7-dWdnZ2d@giganews.com>

On Tue, 06 Dec 2016 01:14:35 +0100, Bernd Nawothnig wrote:

> On 2016-12-05, Wildman wrote:
>> And I am trying to write it without using external programs, where
>> possible.
> 
> That is not the Unix way.

Yes, but it is my way.

>> I am a hobby programmer and I've been trying to learn python
>> for a few months now.  The program is 'worthlessware' but it
>> is a 'learning experience' for me.  
> 
> It looks for me like a worthless learning experience.

It is sad that you consider learning something new to
be worthless.  I used the term "worthlessware" in an
economical sense, meaning it has little or no commercial
value.  However, from a learning standpoint I consider
it to be priceless.

>> A friend wrote a similar program in Bash script and I have been
>> working on translating it to Python.
> 
> Stay with shell script for such tasks. It is never a good idea to
> choose the programming language before closer evaluating the problem.

You have a right to your opinion but I fail to see what
that has to do with the price of eggs.  I picked Python
just because I wanted to learn it not because I had a
particular problem to solve.  If your job was to advocate
Python, I would suggest you find another line of work.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From torriem at gmail.com  Tue Dec  6 11:33:53 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 09:33:53 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26fo7$a7s$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
Message-ID: <1e112d8d-d8bf-426f-8b0f-d4e8e82db23d@gmail.com>

On 12/06/2016 06:52 AM, BartC wrote:
> But those would be silly.

But why?

> Some special syntax is known about: | < and > for example. % less so 
> (I've never, ever used it in live input AFAIK).

Yup and why would you think the ? * special syntax is not known about or
should be known about?  Very strange that you would treat them so
differently.

By the way I use %% environment variable expansion all the time in
Windows.  In fact I most use it in file open dialog boxes or in the Run
dialog.  If I want to see my home folder in a hurry, I'll type Win-R and
then put "%userprofile%" in the box and hit enter. Very convenient.  For
me it's faster to then to browse through explorer and pick folders.
Also it should work regardless of locale, and even if folder names are
localized.

> This auto-expansion causes so many problems, places so many restrictions 
> on what can be done, that it doesn't make sense. Auto-expanding 
> parameters is such a big deal, that it should not be the default 
> behaviour; it needs something to tell the command processor to expand.

Yet you seem to be unable to see that applications doing their own
expansion can also cause problems and restrictions and even worse,
there's not a darn thing you as a user can do about it.

> 
> Then you don't get utterly ridiculous and dangerous behaviour such as 
> the cp example Paul Moore came up with (that trumps most of mine actually):

It's potentially dangerous agreed.  So are lots of commands like rm -rf
/ (which some shells will ask you about).  If you understand a few basic
rules of expansion, you can understand easily what happened or would
happen.  I'm not sure but I think many distros by default in the shell
profiles alias cp="cp -i" and rm="rm -i" to help newbies.  I know the
root account has those aliases by default. I'm pretty sure I've disabled
those aliases for my personal user account because they get in the way
of a lot of my common operations.

Again, I point out that these behaviors can be changed and altered by
the user if he so desires. Right at the shell level.  Instead of having
to alter applications themselves that aren't too smart about things.





From torriem at gmail.com  Tue Dec  6 11:35:40 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 09:35:40 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <20161206075148.45f3310b@bigbox.christie.dr>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <20161206075148.45f3310b@bigbox.christie.dr>
Message-ID: <47cbed94-f8ab-0177-5468-22bac1142db4@gmail.com>

On 12/06/2016 06:51 AM, Tim Chase wrote:
> Based on the OP's description, this is a small part of a much larger
> program.  And I would personally rather maintain a large Python
> code-base than a large Bash code-base.

Absolutely.  Especially when you consider inxi is 12,000 lines of bash
code in one file. Shudder.  Though I'm sure bash made it very easy to
interact with dozens or even hundreds of different external programs on
dozens of operating systems and distributions to gather information.



From bc at freeuk.com  Tue Dec  6 11:41:37 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 16:41:37 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.87.1481039103.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
Message-ID: <o26pll$fpk$1@dont-email.me>

On 06/12/2016 15:44, Michael Torrie wrote:
> On 12/06/2016 04:43 AM, BartC wrote:

> Yes shell expansion has it's gotchas.  But those can all be learned,

Yes, learn to avoid wildcards in command parameters at all costs. But we 
both know that is not satisfactory.

And it's not much help when someone types in:

  program *

and the program has to try and clean up the mess, if it can see it is a 
mess. But remember:

   cp *.c

There might be some irate users out there if it can't detect a simple 
user error like that.

> whereas it's much harder to learn and remember all the gotchas and bugs
> of many individual applications' unique ways of dealing with globs. I'd
> rather deal with shells.
>

OK, I understand. Having a program launcher indiscriminately transform 
input A into A' is /much/ better than having it done by the program, 
even if the program doesn't want it and it is not meaningful.

The fact that you also lose format information (is it three parameters, 
or one parameter transformed into three) is an extra bonus.

This is clearly much better than any other scheme because:

(1) It's Unix not Windows; everything in Unix is always better and 
always make sense.

(2) There have been 40 years of it working this way and there have never 
been any problems. (That is, 40 years of writing programs with 
stultified command line interfaces to make sure that is always the case. 
[As for 'problems' such as the 'cp *.c' one, that's a feature!]

-- 
Bartc


From torriem at gmail.com  Tue Dec  6 11:45:05 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 09:45:05 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <GbmdnUBEet5TedvFnZ2dnUU7-dWdnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me> <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <GbmdnUBEet5TedvFnZ2dnUU7-dWdnZ2d@giganews.com>
Message-ID: <7ef5bad7-4b3d-15db-f4c9-1edd8e5a4013@gmail.com>

On 12/06/2016 09:18 AM, Wildman via Python-list wrote:
> It is sad that you consider learning something new to
> be worthless.  I used the term "worthlessware" in an
> economical sense, meaning it has little or no commercial
> value.  However, from a learning standpoint I consider
> it to be priceless.

Well said. Some of us get hung up so much on the proper way to do
something that we end up not doing much at all, other than talk about
the proper way to do things.  I tend to have this problem.  While I
talked about the proper and theoretical ways of doing agricultural GPS
coverage mapping, another person with less formal programming training
than I started hacking and you know what? He has a functioning program
now that actually works. It's in C# (which I don't love), and it's got
rough spots in the code and it's a bit difficult to add certain features
to, but he simply went and did and learned as he went.  Now he's at the
point where he could refactor (and do it quickly) to get the
architecture a bit more robust.  But the point is I wasted all my time
thinking about how I might do it and he just did it.  Was very
instructive to me.

>>> A friend wrote a similar program in Bash script and I have been
>>> working on translating it to Python.
>>
>> Stay with shell script for such tasks. It is never a good idea to
>> choose the programming language before closer evaluating the problem.
> 
> You have a right to your opinion but I fail to see what
> that has to do with the price of eggs.  I picked Python
> just because I wanted to learn it not because I had a
> particular problem to solve.  If your job was to advocate
> Python, I would suggest you find another line of work.

I appreciate your measured response to what could be seen as an
inflammatory post.

Sometimes I think it all depends on the purpose for which you do
something. In this case it's for fun, so knock yourself out. If you were
instead writing this as part of a requirement for some enterprise server
process professionally, you'd probably want to stick with Bash rather
than shoe-horn Python into a systems programming language and
shell-scripting language, which it's not really that good at.  I can say
this given my professional experience with server shell scripting.




From python at mrabarnett.plus.com  Tue Dec  6 11:53:30 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 6 Dec 2016 16:53:30 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CACL+1avD311T8f8Pgb6LkQhYZjO5cCoBmNG4UXu2+qEMXPg9Tg@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <ean6fiF6nc9U1@mid.individual.net>
 <o268uu$ig8$1@dont-email.me>
 <CAPTjJmqoq1CnAshS4zsWNwQHnDg7mUMjXR6J2Jt5PGnmDuvgvA@mail.gmail.com>
 <CACL+1avD311T8f8Pgb6LkQhYZjO5cCoBmNG4UXu2+qEMXPg9Tg@mail.gmail.com>
Message-ID: <5708aba0-9db6-6e39-c060-e2aece40949d@mrabarnett.plus.com>

On 2016-12-06 13:08, eryk sun wrote:
> On Tue, Dec 6, 2016 at 12:26 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
>>> In that directory (which was on Windows but accessible via a virtual Linux),
>>> typing any Linux command followed by * would have required all 3.4 million
>>> directory entries to be accessed in order to build a 3.4 million-element
>>> argv list. I've no idea how long that would have taken.
>>
>> I just asked Python to build me a 4-million-element list, and it took
>> no visible time - a small fraction of a second. Don't be afraid of
>> large argument lists. We're not writing 8088 Assembly Language
>> programs in 64KB of working memory here.
>
> The problem isn't building an arbitrary list with millions of
> elements. The problem is the time it would take to read millions of
> filenames from a directory. It depends on the performance of the disk
> and filesystem. Be careful with globbing. Think about the consequences
> before running a command, especially if you're in the habit of
> creating directories with hundreds of thousands, or millions, of
> files. It's not a problem that I've ever had to deal with.
>
Many years ago I was working with a database application running on 
MSDOS that stored predefined queries in files, one query per file. There 
were many queries (though fewer than a thousand), resulting in many 
small files in a single directory. Fetching one of those predefined 
queries was surprisingly slow.

I found that it was a lot faster to put them into a single file and then 
call an external program to extract the one wanted. It also took up a 
lot less disk space!



From python at mrabarnett.plus.com  Tue Dec  6 12:00:20 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 6 Dec 2016 17:00:20 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26fo7$a7s$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
Message-ID: <2da57b90-bf62-ef6c-a9ee-dbe5318744d6@mrabarnett.plus.com>

On 2016-12-06 13:52, BartC wrote:
> On 06/12/2016 13:34, Chris Angelico wrote:
>> On Wed, Dec 7, 2016 at 12:25 AM, BartC <bc at freeuk.com> wrote:
>>> What, the convention of NOT assuming that any command parameter that uses *
>>> or ? MUST refer to whatever set of filenames happen to match in the current
>>> directory? And to then insert N arbitrary filenames in the parameter list.
>>>
>>> That's a pretty good convention, yes?!
>>
>> Right! And while you're at it, stop assuming that percent signs have
>> meaning, that quotes have meaning, etc, etc, etc. Right? And why
>> should the enter key be significant - what if you want to send more
>> than one line of command line arguments?
>
> But those would be silly.
>
> Some special syntax is known about: | < and > for example. % less so
> (I've never, ever used it in live input AFAIK).
>
[snip]

You've never used ImageMagick?

If you want to shrink an image to half its size:

     convert dragon.gif -resize 50% half_dragon.gif



From bc at freeuk.com  Tue Dec  6 12:36:23 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 17:36:23 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.92.1481043627.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
 <2da57b90-bf62-ef6c-a9ee-dbe5318744d6@mrabarnett.plus.com>
 <mailman.92.1481043627.2253.python-list@python.org>
Message-ID: <o26ssb$sde$1@dont-email.me>

On 06/12/2016 17:00, MRAB wrote:
> On 2016-12-06 13:52, BartC wrote:

>> Some special syntax is known about: | < and > for example. % less so
>> (I've never, ever used it in live input AFAIK).
>>
> [snip]
>
> You've never used ImageMagick?
>
> If you want to shrink an image to half its size:
>
>     convert dragon.gif -resize 50% half_dragon.gif

No. But that '50%' is seen by apps (eg. with Python's 'print sys.argv') 
on both Windows and Linux.

However, I can imagine that a calculator app might have trouble with 
some expressions:

    calculate 3*5

This expression might be seen as 345 if there happens to be file called 
'345' lying around.

-- 
Bartc


From lew.pitcher at digitalfreehold.ca  Tue Dec  6 13:00:36 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Tue, 06 Dec 2016 13:00:36 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
 <2da57b90-bf62-ef6c-a9ee-dbe5318744d6@mrabarnett.plus.com>
 <mailman.92.1481043627.2253.python-list@python.org>
 <o26ssb$sde$1@dont-email.me>
Message-ID: <o26u9l$24m$1@dont-email.me>

On Tuesday December 6 2016 12:36, in comp.lang.python, "BartC" <bc at freeuk.com>
wrote:

> On 06/12/2016 17:00, MRAB wrote:
>> On 2016-12-06 13:52, BartC wrote:
> 
>>> Some special syntax is known about: | < and > for example. % less so
>>> (I've never, ever used it in live input AFAIK).
>>>
>> [snip]
>>
>> You've never used ImageMagick?
>>
>> If you want to shrink an image to half its size:
>>
>>     convert dragon.gif -resize 50% half_dragon.gif
> 
> No. But that '50%' is seen by apps (eg. with Python's 'print sys.argv')
> on both Windows and Linux.
> 
> However, I can imagine that a calculator app might have trouble with
> some expressions:
> 
>     calculate 3*5

I can't see this being an issue with the "calculator app", unless the
calculator app itself is written to perform file globbing.

It /might/ be an issue with the shell, if you permit it to glob
the "calculator app" arguments before invoking the "calculator app" binary.
But, then again, Unix shell filename globbing is something that the user can
disable temporarily or permanently if necessary.

For example:
  calculate '3*5'
or
  sh -o noglob -c "calculate 3*5"
or even
  sh -o noglob
  calculate 3*5

> This expression might be seen as 345 if there happens to be file called
> '345' lying around.

Only if shell globbing is enabled, and you don't specifically bypass it.


-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From best_lay at yahoo.com  Tue Dec  6 13:10:52 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 06 Dec 2016 12:10:52 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
Message-ID: <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>

On Mon, 05 Dec 2016 16:08:57 -0600, Tim Chase wrote:

> On 2016-12-05 14:58, Wildman via Python-list wrote:
>> I there a way to detect what the Linux runlevel is from
>> within a Python program?  I would like to be able to do
>> it without the use of an external program such as 'who'
>> or 'runlevel'.
> 
> You can use something like
> 
> https://gist.github.com/likexian/f9da722585036d372dca

I went back to the code from the above link to try to
get it to work with Python3, just to see if I could.
The problem was that the output was in bytes or partly
in bytes like this:

['1', '53', "b'~\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\
x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ (...)

I was trying to convert the bytes to strings and that
is what I never got to work right.  It didn't occur
to me that all I needed was the first two fields and
they were already strings.

The 'print output' part of the original code was this
which printed everything.  Over kill for my use:

data = read_xtmp('/var/run/utmp')
for i in data:
    print i

I changed it to this and it works:

data = read_xtmp('/var/run/utmp')
for i in data:
    if i[0] == "1":
        print("Runlevel: " + chr(int(i[1]) & 0xFF))
        break

The output:  Runlevel: 5

If I had tried this in the beginning, it would have
save you a lot of work.

Since both versions of the code works, which one do
you recommend?  Or does it matter?

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From walters.justin01 at gmail.com  Tue Dec  6 13:17:34 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Tue, 6 Dec 2016 10:17:34 -0800
Subject: When will they fix Python _dbm?
In-Reply-To: <w4o957e3n2v24639mwec7vrj.1480986370390@email.android.com>
References: <w4o957e3n2v24639mwec7vrj.1480986370390@email.android.com>
Message-ID: <CAO1D73Ek2h6MX0m-SBP+3nmMdGbb1CJDZfZxCkA2Z9C0K=sisw@mail.gmail.com>

On Mon, Dec 5, 2016 at 5:06 PM, clvanwall <clvanwall at gmail.com> wrote:

> will thid do?  John
>

Looks like you need to use dbm.ndbm.open() instead of just dbm.open().

See the docs here for more info:
https://docs.python.org/3/library/dbm.html#module-dbm.ndbm


From walters.justin01 at gmail.com  Tue Dec  6 13:28:36 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Tue, 6 Dec 2016 10:28:36 -0800
Subject: Django broken pipe error
In-Reply-To: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
References: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
Message-ID: <CAO1D73FD5zxqOY9ogckDgAKtJwbvL-MkFAZbjS1KqYo_89G1Hw@mail.gmail.com>

On Tue, Dec 6, 2016 at 6:21 AM, <dr.roman.graf at gmail.com> wrote:

> Hi,
>
> I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that
> apparently is a not fixed Django bug. Does anybody now how to fix it? I've
> been searching a lot and didn't find any solution.
>
> This error happens very irregularly by Post request in Django. Sometimes
> it works sometimes not:
>
> [06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/SocketServer.py", line 593, in
> process_request_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-
> packages/django/core/servers/basehttp.py", line 129, in __init__
>     super(WSGIRequestHandler, self).__init__(*args, **kwargs)
>   File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
>     self.finish()
>   File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
>     self.wfile.close()
>   File "/usr/lib/python2.7/socket.py", line 279, in close
>     self.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
>     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> ----------------------------------------
> Exception happened during processing of request from ('127.0.0.1', 38224)
>
> It is also described in: https://www.reddit.com/r/
> joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/
>
> On https://bugs.python.org/issue14574 is stated that this error should
> already be fixed but apparently not.
>
> Best regards,
>
> Roman


Hi Roman,

Is this happening on the dev server or a production server? Did you use the
command ./manage.py runserver to start the server?

Can you please provide the code for the view class/function that is
throwing the error? I believe that this issue
may be caused by a call to the database not being structured correctly.


From python.list at tim.thechases.com  Tue Dec  6 14:06:35 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Tue, 6 Dec 2016 13:06:35 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>
Message-ID: <20161206130635.260c5816@bigbox.christie.dr>

On 2016-12-06 12:10, Wildman via Python-list wrote:
> If I had tried this in the beginning, it would have
> save you a lot of work.
> 
> Since both versions of the code works, which one do
> you recommend?  Or does it matter?

Heh, I'm not sure it matters much.  The code I provided should be
expandable for tidily handling other entries in utmp, allowing you to
search for other system events that might interest you (check out the
invocation of the filter() function which filters by type).

Since I've already done the leg-work, I mildly advocate using my
version since it should be pretty clean and easy to expand.  But if
you want the satisfaction of using your code, I won't take offense :-)

-tkc





From best_lay at yahoo.com  Tue Dec  6 14:27:42 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 06 Dec 2016 13:27:42 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>
 <20161206130635.260c5816@bigbox.christie.dr>
 <mailman.95.1481051221.2253.python-list@python.org>
Message-ID: <oYSdnSixw7GzjNrFnZ2dnUU7-fmdnZ2d@giganews.com>

On Tue, 06 Dec 2016 13:06:35 -0600, Tim Chase wrote:

> On 2016-12-06 12:10, Wildman via Python-list wrote:
>> If I had tried this in the beginning, it would have
>> save you a lot of work.
>> 
>> Since both versions of the code works, which one do
>> you recommend?  Or does it matter?
> 
> Heh, I'm not sure it matters much.  The code I provided should be
> expandable for tidily handling other entries in utmp, allowing you to
> search for other system events that might interest you (check out the
> invocation of the filter() function which filters by type).

Yes, your code is very expandable and I will keep it
archived for that reason.  In the future I might want
to delve further into utmp.  But right now all I need
is the runlevel.

> Since I've already done the leg-work, I mildly advocate using my
> version since it should be pretty clean and easy to expand.  But if
> you want the satisfaction of using your code, I won't take offense :-)
> 
> -tkc

Yes, I think I will use my code.  No offense intended. :-)
Again, I do appreciate all your work on my behalf.  I hope
some day I can return the favor.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From greg.ewing at canterbury.ac.nz  Tue Dec  6 16:28:03 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Wed, 07 Dec 2016 10:28:03 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26e5l$4k6$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
Message-ID: <eaoor5Fiuu3U1@mid.individual.net>

BartC wrote:
> What, the convention of NOT assuming that any command parameter that 
> uses * or ? MUST refer to whatever set of filenames happen to match in 
> the current directory?

Yes.

> That's a pretty good convention, yes?!

That's a matter of opinion. It precludes the shell from
performing various services that Unix users like their
shells to provide.

> Or someone from Britain visiting the USA and saying OMG, everyone's got 
> a gun! Suppose someone runs amok and shoots everybody!

If you expect us to trust most Americans not to misuse
their guns, you will appreciate us asking you to trust
most Unix users not to misuse their wildcard patterns.

-- 
Greg


From ian.g.kelly at gmail.com  Tue Dec  6 16:34:15 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 6 Dec 2016 14:34:15 -0700
Subject: When will they fix Python _dbm?
In-Reply-To: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
References: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
Message-ID: <CALwzidmCN6Nhc456YJ_5XiDqDu78PENZhr0vjLNaqW+w7VpVqg@mail.gmail.com>

On Mon, Dec 5, 2016 at 7:45 AM, clvanwall <clvanwall at gmail.com> wrote:
> I have been a Perl programmer for 15+ years and decided to give Python a try.  My platform is windows and I installed the latest 3.5.2. Next I decided to convert a perl program that uses a ndbm database since according to the doc on python, it should be able to work with it.  Needless to say, I get: dbm.error: db type is dbm.ndbm, but the module is not available
> Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!  That's a long time for a bug to be left open.

Are you referring to http://bugs.python.org/issue14185? That's on
Linux platforms and it has to do with building Python, not using it.

The dbm.ndbm documentation specifically says that it provides an
interface to the *Unix* ndbm library:
https://docs.python.org/3/library/dbm.html#module-dbm.ndbm. I don't
think that this module is part of the standard Windows distribution of
Python. You might be able to find a third-party distribution of the
module using a Windows port of ndbm, or you could try to build it
yourself.


From greg.ewing at canterbury.ac.nz  Tue Dec  6 16:44:05 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Wed, 07 Dec 2016 10:44:05 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26fo7$a7s$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
Message-ID: <eaopp7Fj5v8U1@mid.individual.net>

BartC wrote:
> But those would be silly.
> 
> Some special syntax is known about: | < and > for example. % less so

What you need to understand is that, to a Unix user,
* and ? are *just as well known* as |, < and >. Perhaps
even more so, because they're likely to be used much
sooner than piping and redirection. And when learning
about them, the fact that they're interpreted by the
shell is learned at the same time.

> And the justification? Well, %ENVIRONMENTVARIABLE% gets converted in 
> Windows, so why not?!

No, the justification is that the Unix convention allows
the shell to provide certain useful functions that Unix
users value.

If you don't want those functions, you're free to write
your own shell that works however you want. Complaining
that everyone *else* should want the same things you
want is not reasonable.

-- 
Greg


From cs at zip.com.au  Tue Dec  6 16:46:28 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 7 Dec 2016 08:46:28 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o26pll$fpk$1@dont-email.me>
References: <o26pll$fpk$1@dont-email.me>
Message-ID: <20161206214628.GA83577@cskk.homeip.net>

On 06Dec2016 16:41, BartC <bc at freeuk.com> wrote:
>On 06/12/2016 15:44, Michael Torrie wrote:
>>On 12/06/2016 04:43 AM, BartC wrote:
>>Yes shell expansion has it's gotchas.  But those can all be learned,
>
>Yes, learn to avoid wildcards in command parameters at all costs. But 
>we both know that is not satisfactory.

Sigh. I'm sure this has all been thrashed out in this huge thread, but: that is 
what quoting is for (in the shell): to control whether wildcards are expanded 
or not. You, the typist, get to decide.

>And it's not much help when someone types in:
>
> program *
>
>and the program has to try and clean up the mess, if it can see it is 
>a mess.

Some invocations will be nonsense, and a program may catch those. But if that 
was actually the typist's intent, and the program says "nah!"?

>But remember:
>
>  cp *.c
>
>There might be some irate users out there if it can't detect a simple 
>user error like that.

If there are _2_ .c files present, that will indeed misbehave. But often there 
are several, and cp detects the "copy several things to one target" mode for:

  cp a.c b.c d.c target-dir

and says the final target (eg "d.c") is not a directory. A degree of safety.  
This is the circumstance where the request, as it is received, is nonsense and 
detectably so. Not perfectly robust, but you can never be perfectly robust 
against the typist making a mistaken request, globbing or not.

>>whereas it's much harder to learn and remember all the gotchas and bugs
>>of many individual applications' unique ways of dealing with globs. I'd
>>rather deal with shells.
>
>OK, I understand. Having a program launcher indiscriminately transform 
>input A into A' is /much/ better than having it done by the program, 
>even if the program doesn't want it and it is not meaningful.

It isn't indiscriminate; quoting lets the typist discriminate. Michael's point 
is that at least you're always using the _same_ command line conventions about 
what shall be expanded and what shall not, rather than a rule per app.

>The fact that you also lose format information (is it three parameters, or one 
>parameter transformed into three) is an extra bonus.

Feh. If *.c matches 3 files and I didn't quote, it is 3 parameters in my mind.  
If I wanted one parameter I'd have said '*.c' (<== quotes).
>
>This is clearly much better than any other scheme because:
>
>(1) It's Unix not Windows; everything in Unix is always better and 
>always make sense.

UNIX rules are often simpler and more consistent.

>(2) There have been 40 years of it working this way and there have 
>never been any problems. (That is, 40 years of writing programs with 
>stultified command line interfaces to make sure that is always the case. [As 
>for 'problems' such as the 'cp *.c' one, that's a feature!]

Nothing prevents you writing an extremely simple shell yourself you know. It 
needn't expand anything. _Or_ you could have it adopt the inverse convention: 
expand nothing unless asked. Eg:

  cp G:*.c

to cause "*.c" to get expanded. Of course, because the various executables 
don't feel any burden to implement globbing you may run into some impedence 
mismatch.

Cheers,
Cameron Simpson <cs at zip.com.au>


From rosuav at gmail.com  Tue Dec  6 17:24:33 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 7 Dec 2016 09:24:33 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <20161206214628.GA83577@cskk.homeip.net>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
Message-ID: <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>

On Wed, Dec 7, 2016 at 8:46 AM, Cameron Simpson <cs at zip.com.au> wrote:
> Nothing prevents you writing an extremely simple shell yourself you know. It
> needn't expand anything. _Or_ you could have it adopt the inverse
> convention: expand nothing unless asked. Eg:
>
>  cp G:*.c
>
> to cause "*.c" to get expanded. Of course, because the various executables
> don't feel any burden to implement globbing you may run into some impedence
> mismatch.

I actually have something like this in one application's inbuilt
command executor - it does globbing ONLY if the parameter starts with
a question mark (eg "?*.c" will glob "*.c" in that directory). It's
deliberate, but frankly, it ends up annoying more often than not. I'm
considering switching to a more normal way of doing it.

ChrisA


From marko at pacujo.net  Tue Dec  6 17:29:00 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 00:29:00 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
Message-ID: <87d1h4pu8j.fsf@elektro.pacujo.net>

Tim Chase <python.list at tim.thechases.com>:

> This works based on my poking at it in both Py2 and Py3:

Great info, Tim.

A word a warning: your code doesn't lock /var/run/utmp before access,
which is a race condition. The file may be updated at any time, and
ordinary file reads may yield corrupted records.

The library functions getutline(), pututline() etc lock the file
internally (although that is not documented). You can see the (advisory
record) locking in action by running the command

   strace who


Since the locking scheme hasn't been documented, the only safe way to
read /var/run/utmp is through the C API functions.

Another thing is that, as stated before, the runlevel business is
legacy. It is still supported by systemd-update-utmp, but for how long
is anybody's guess.


Marko


From nathan.ernst at gmail.com  Tue Dec  6 17:35:20 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Tue, 6 Dec 2016 16:35:20 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
Message-ID: <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>

One other consideration in regards to globbing in the argument list:
there's a static limit to the byte length of argv. On windows, it's 8191
bytes (I'm assuming a null-terminator brings that to 8192, which is a weird
2**13). For Linux, as of kernal 2.6.25, apparently, the limit is 131072
bytes, an equally odd choice of 2**17 bytes.

I'm not sure if globs directly to commands bypass these restrictions, but
I've seen real world issues with build systems with thousands of files that
attempted to pass all dependencies to a target that blew up spectacularly
(one of the reasons cl & link on windows accept argument files as input
instead of just command line arguments).

Regards,
Nate

On Tue, Dec 6, 2016 at 4:24 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Dec 7, 2016 at 8:46 AM, Cameron Simpson <cs at zip.com.au> wrote:
> > Nothing prevents you writing an extremely simple shell yourself you
> know. It
> > needn't expand anything. _Or_ you could have it adopt the inverse
> > convention: expand nothing unless asked. Eg:
> >
> >  cp G:*.c
> >
> > to cause "*.c" to get expanded. Of course, because the various
> executables
> > don't feel any burden to implement globbing you may run into some
> impedence
> > mismatch.
>
> I actually have something like this in one application's inbuilt
> command executor - it does globbing ONLY if the parameter starts with
> a question mark (eg "?*.c" will glob "*.c" in that directory). It's
> deliberate, but frankly, it ends up annoying more often than not. I'm
> considering switching to a more normal way of doing it.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From rosuav at gmail.com  Tue Dec  6 17:46:26 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 7 Dec 2016 09:46:26 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
Message-ID: <CAPTjJmr04vwq9Aq8goJtCO5smoyhQtKsJMVpeqvjJQawGsnKzw@mail.gmail.com>

On Wed, Dec 7, 2016 at 9:35 AM, Nathan Ernst <nathan.ernst at gmail.com> wrote:
> One other consideration in regards to globbing in the argument list: there's
> a static limit to the byte length of argv. On windows, it's 8191 bytes (I'm
> assuming a null-terminator brings that to 8192, which is a weird 2**13). For
> Linux, as of kernal 2.6.25, apparently, the limit is 131072 bytes, an
> equally odd choice of 2**17 bytes.
>
> I'm not sure if globs directly to commands bypass these restrictions, but
> I've seen real world issues with build systems with thousands of files that
> attempted to pass all dependencies to a target that blew up spectacularly
> (one of the reasons cl & link on windows accept argument files as input
> instead of just command line arguments).

A lot of things are a problem in an ecosystem they're not designed
for, but work fine if you have them. If the normal thing is to have
the shell expand globs, the system will expect and assume this,
applications will expect and assume this, and the shell will do this.
If the normal thing is not to, then nobody will expect it. This is
only going to be a problem when you mix ecosystems; for instance, I
don't know what the Win 10 internal bash does, with a limit like that,
and I don't know how Wine apps cope with file names that include
metacharacters in them. But when everything's built consistently, you
don't have a problem.

ChrisA


From breamoreboy at gmail.com  Tue Dec  6 17:54:03 2016
From: breamoreboy at gmail.com (breamoreboy at gmail.com)
Date: Tue, 6 Dec 2016 14:54:03 -0800 (PST)
Subject: When will they fix Python _dbm?
In-Reply-To: <mailman.96.1481060104.2253.python-list@python.org>
References: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
 <CALwzidmCN6Nhc456YJ_5XiDqDu78PENZhr0vjLNaqW+w7VpVqg@mail.gmail.com>
 <mailman.96.1481060104.2253.python-list@python.org>
Message-ID: <e0eb9357-a437-4dfd-bb63-d48932765e10@googlegroups.com>

On Tuesday, December 6, 2016 at 9:35:19 PM UTC, Ian wrote:
> On Mon, Dec 5, 2016 at 7:45 AM, clvanwall wrote:
> > I have been a Perl programmer for 15+ years and decided to give Python a try.  My platform is windows and I installed the latest 3.5.2. Next I decided to convert a perl program that uses a ndbm database since according to the doc on python, it should be able to work with it.  Needless to say, I get: dbm.error: db type is dbm.ndbm, but the module is not available
> > Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!  That's a long time for a bug to be left open.
> 
> Are you referring to http://bugs.python.org/issue14185? That's on
> Linux platforms and it has to do with building Python, not using it.
> 
> The dbm.ndbm documentation specifically says that it provides an
> interface to the *Unix* ndbm library:
> https://docs.python.org/3/library/dbm.html#module-dbm.ndbm. I don't
> think that this module is part of the standard Windows distribution of
> Python. You might be able to find a third-party distribution of the
> module using a Windows port of ndbm, or you could try to build it
> yourself.

How about http://www.gnu.org/software/gdbm/gdbm.html

Kindest regards.

Mark Lawrence.


From torriem at gmail.com  Tue Dec  6 17:58:00 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 15:58:00 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
Message-ID: <f7ec6625-a1f6-225f-5c18-9700bc8a5484@gmail.com>

On 12/06/2016 03:35 PM, Nathan Ernst wrote:
> One other consideration in regards to globbing in the argument list:
> there's a static limit to the byte length of argv. On windows, it's 8191
> bytes (I'm assuming a null-terminator brings that to 8192, which is a weird
> 2**13). For Linux, as of kernal 2.6.25, apparently, the limit is 131072
> bytes, an equally odd choice of 2**17 bytes.
> 
> I'm not sure if globs directly to commands bypass these restrictions, but
> I've seen real world issues with build systems with thousands of files that
> attempted to pass all dependencies to a target that blew up spectacularly
> (one of the reasons cl & link on windows accept argument files as input
> instead of just command line arguments).

Applications that can do their own globbing do avoid this command-line
length limit for sure, at least in these situations.  Instead of
excepting one or more filenames on the command-line, it could accept a
filespec, a single field.  So one million files would just be "foo *.*".
 This is part of what Bart really likes about the windows way of doing
things. And it is an advantage of that scheme.

Anyway, I'm interested to hear Bart's opinion of PowerShell, which is
set to be the default shell for the Windows Console starting very soon.
cmd.exe will still be available for a while, but I imagine in a couple
of windows versions it will disappear entirely.  I've little experience
with PowerShell, so I don't know whether it will bring about a new level
of argument interpretation that will get Bart all upset or if it will be
the cat's meow.  Some parts of it look awfully unix-like such as cat and
nice looping constructs. Other parts bear little resemblance to anything
I've used before.  Seems a little verbose to me (it is a shell that
wraps a complete object system).




From torriem at gmail.com  Tue Dec  6 18:00:53 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 16:00:53 -0700
Subject: Detect Linux Runlevel
In-Reply-To: <87d1h4pu8j.fsf@elektro.pacujo.net>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
Message-ID: <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>

On 12/06/2016 03:29 PM, Marko Rauhamaa wrote:
> Another thing is that, as stated before, the runlevel business is
> legacy. It is still supported by systemd-update-utmp, but for how long
> is anybody's guess.

System V compatibility is still important to Linux, and as long as it
is, something resembling a runlevel has to be provided, even it's just
an abstraction of something else like a systemd target.  So any system
that advertises System V compliance or compatibility will have a runlevel.



From bc at freeuk.com  Tue Dec  6 18:37:48 2016
From: bc at freeuk.com (BartC)
Date: Tue, 6 Dec 2016 23:37:48 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <eaopp7Fj5v8U1@mid.individual.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me> <eaopp7Fj5v8U1@mid.individual.net>
Message-ID: <o27i21$gqt$1@dont-email.me>

On 06/12/2016 21:44, Gregory Ewing wrote:
> BartC wrote:

>> And the justification? Well, %ENVIRONMENTVARIABLE% gets converted in
>> Windows, so why not?!
>
> No, the justification is that the Unix convention allows
> the shell to provide certain useful functions that Unix
> users value.
>
> If you don't want those functions, you're free to write
> your own shell that works however you want. Complaining
> that everyone *else* should want the same things you
> want is not reasonable.

How does that work?

Suppose I provide an assortment of applications that would work better 
if wildcards are expanded.

Do I then have to provide one more application, a shell, to be run first 
if someone wants to run any of my applications? Which they then have to 
quit if they want to run some other programs that depend on wildcard 
expansion.

(Actually at this point I haven't got a clue as to how Unix applications 
are distributed. I guess it's not as simple as just providing a binary 
executable. For the moment, I'm using C source code as every Unix system 
has a C compiler. I suppose it could be Python source too, but I doubt 
if my interpreters written in Python will run quite as briskly -- I find 
C slow enough for this purpose.)

-- 
Bartc


From torriem at gmail.com  Tue Dec  6 19:18:40 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 6 Dec 2016 17:18:40 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o27i21$gqt$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me> <eaopp7Fj5v8U1@mid.individual.net>
 <o27i21$gqt$1@dont-email.me>
Message-ID: <d9b11523-c74a-3171-37d0-ab6083c9f0bd@gmail.com>

On 12/06/2016 04:37 PM, BartC wrote:
> How does that work?
> 
> Suppose I provide an assortment of applications that would work better 
> if wildcards are expanded.
> 
> Do I then have to provide one more application, a shell, to be run first 
> if someone wants to run any of my applications? Which they then have to 
> quit if they want to run some other programs that depend on wildcard 
> expansion.
> 
> (Actually at this point I haven't got a clue as to how Unix applications 
> are distributed. I guess it's not as simple as just providing a binary 
> executable. For the moment, I'm using C source code as every Unix system 
> has a C compiler. I suppose it could be Python source too, but I doubt 
> if my interpreters written in Python will run quite as briskly -- I find 
> C slow enough for this purpose.)

I'm not sure what your point is. If your app targets Windows, you'll
need to support the filespec thing and do your own globbing because
that's the only mechanism Windows provides.  If your app targets unix,
you'll probably want to do things the way unix users expect. If unix
targets are important to you you take the time to learn about the
environment in which you want your programs to run.



From ian.g.kelly at gmail.com  Tue Dec  6 19:52:03 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 6 Dec 2016 17:52:03 -0700
Subject: When will they fix Python _dbm?
In-Reply-To: <CALwzidm0FKLjJGtUxuTS4Rhvd1G+dExU8WuUrGe8Y71L8KVkkg@mail.gmail.com>
References: <w7dw4xm1rl3uboth0j06r43o.1480949153380@email.android.com>
 <CALwzidmCN6Nhc456YJ_5XiDqDu78PENZhr0vjLNaqW+w7VpVqg@mail.gmail.com>
 <mailman.96.1481060104.2253.python-list@python.org>
 <e0eb9357-a437-4dfd-bb63-d48932765e10@googlegroups.com>
 <CALwzid=UkCe_R-abM3=Cn505HgxdP_WonhwJ3YdfYW7qduYqyQ@mail.gmail.com>
 <CALwzidm0FKLjJGtUxuTS4Rhvd1G+dExU8WuUrGe8Y71L8KVkkg@mail.gmail.com>
Message-ID: <CALwzidkyhaqSr=2WdCme_JPgqii59HRtb6Yn1+W=UdL=Y3vSXw@mail.gmail.com>

On Dec 6, 2016 4:04 PM, <breamoreboy at gmail.com> wrote:

On Tuesday, December 6, 2016 at 9:35:19 PM UTC, Ian wrote:
> On Mon, Dec 5, 2016 at 7:45 AM, clvanwall wrote:
> > I have been a Perl programmer for 15+ years and decided to give Python
a try.  My platform is windows and I installed the latest 3.5.2. Next I
decided to convert a perl program that uses a ndbm database since according
to the doc on python, it should be able to work with it.  Needless to say,
I get: dbm.error: db type is dbm.ndbm, but the module is not available
> > Searching on Python Bug Tracker shows _dbm missing back in 03-03-2012!
That's a long time for a bug to be left open.
>
> Are you referring to http://bugs.python.org/issue14185? That's on
> Linux platforms and it has to do with building Python, not using it.
>
> The dbm.ndbm documentation specifically says that it provides an
> interface to the *Unix* ndbm library:
> https://docs.python.org/3/library/dbm.html#module-dbm.ndbm. I don't
> think that this module is part of the standard Windows distribution of
> Python. You might be able to find a third-party distribution of the
> module using a Windows port of ndbm, or you could try to build it
> yourself.

How about http://www.gnu.org/software/gdbm/gdbm.html


Well, that would be for dbm.gnu, not dbm.ndbm. And I don't see any
distribution of the Python module there in any case.


From python.list at tim.thechases.com  Tue Dec  6 20:00:43 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Tue, 6 Dec 2016 19:00:43 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <87d1h4pu8j.fsf@elektro.pacujo.net>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
Message-ID: <20161206190043.7fc99068@bigbox.christie.dr>

On 2016-12-07 00:29, Marko Rauhamaa wrote:
> Tim Chase <python.list at tim.thechases.com>:
> 
> > This works based on my poking at it in both Py2 and Py3:
> 
> Great info, Tim.
> 
> A word a warning: your code doesn't lock /var/run/utmp before
> access, which is a race condition. The file may be updated at any
> time, and ordinary file reads may yield corrupted records.

Since the code is reading in record-sized blocks and never writing,
I'm not sure how much possibility there is for a race condition. At
worst, I imagine that it would result in reading the old data which
isn't a horrible condition.

For under a certain block-size (PIPE_BUF, which I think used to be
the minimum POSIX requirement of 512b, but is now 4096b on Linux),
*nix operating systems were atomic in their reading and writing.  So
as long as the writer is writing a record at a time (or less) and not
issuing multiple writes to update disjoint parts of the record, I'm
pretty confident that atomicity won't be an issue for all intents and
purposes.

http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html#tag_03_866_08

(though according to my "man 2 write" on my Linux box, before Linux
3.14, it wasn't atomic according to specs)

> The library functions getutline(), pututline() etc lock the file
> internally (although that is not documented). You can see the
> (advisory record) locking in action by running the command

C API functions which don't appear to be exposed in stock Python. ;-)

> Another thing is that, as stated before, the runlevel business is
> legacy. It is still supported by systemd-update-utmp, but for how
> long is anybody's guess.

Much like Unix itself, if the OP chooses to shoot off his own feet
with them, my aim is to do it efficiently as requested. ;-)

-tkc




From bc at freeuk.com  Tue Dec  6 20:55:38 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 01:55:38 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.99.1481063728.2253.python-list@python.org>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
Message-ID: <o27q4d$am8$1@dont-email.me>

On 06/12/2016 22:35, Nathan Ernst wrote:
> One other consideration in regards to globbing in the argument list:
> there's a static limit to the byte length of argv. On windows, it's 8191
> bytes

That's the length of the command line, where parameters might still be 
in the form *.*, ie. pre-splitting and pre-expansion. After converting 
to argv format (which I believe is done by a C runtime function), that 
will be in different memory with presumably its own limits.

> (I'm assuming a null-terminator brings that to 8192, which is a weird
> 2**13). For Linux, as of kernal 2.6.25, apparently, the limit is 131072
> bytes, an equally odd choice of 2**17 bytes.

Maybe they started off at 4KB and 64KB but ran into problems.

But even Linux's 128KB will fill if someone wanted a command line that 
listed 20,000 files individually. But it would be spectacularly bad use 
of a command line interface which was designed for humans.

> I'm not sure if globs directly to commands bypass these restrictions, but
> I've seen real world issues with build systems with thousands of files that
> attempted to pass all dependencies to a target that blew up spectacularly
> (one of the reasons cl & link on windows accept argument files as input
> instead of just command line arguments).

So does gcc on Linux (if you mean the @file syntax). It's a tidier way 
of doing things.

-- 
Bartc


From steve+python at pearwood.info  Tue Dec  6 21:22:09 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 07 Dec 2016 13:22:09 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b8c1d4c1-2611-c950-ca6e-e861f13ca143@gmail.com>
 <mailman.52.1480966148.2253.python-list@python.org>
 <o24k60$ufp$1@dont-email.me> <ean6fiF6nc9U1@mid.individual.net>
Message-ID: <58477252$0$22142$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 06:08 pm, Gregory Ewing wrote:

>> And if there's an error in an option, you may have to abort, which means
>> throwing away that list of files which, in some cases, can run into
>> millions.
> 
> This "millions of files" thing seems to be an imaginary
> monster you've invented to try to scare people. I claim
> that, to a very good approximation, it doesn't exist.
> I've never encountered a directory containing a million
> files, and if such a thing existed, it would be pathological
> in enough other ways to make it a really bad idea.

I didn't read Bart's "millions" as *literally* 2*10**6 or more, I read it
as "a very large number of files". But it is conceivable that it is
millions.

ReiserFS can apparently handle up to approximately 4 billion files in a
single directory; even FAT32 can take up to 268 million files in total,
although there is a per-directory limit of 65535.

Here's a true story of a fellow who ended up with 8 million files in a
single directory:

http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html


There are system dependent limits on globbing expansion and the number of
arguments you can pass to a program:

http://stackoverflow.com/questions/4185017/maximum-number-of-bash-arguments-max-num-cp-arguments

http://www.in-ulm.de/~mascheck/various/argmax/





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Dec  6 21:23:09 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 07 Dec 2016 13:23:09 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
Message-ID: <5847728d$0$22142$c3e8da3$5496439d@news.astraweb.com>

On Tue, 6 Dec 2016 06:37 pm, Larry Hudson wrote:

> On 12/05/2016 10:50 AM, BartC wrote:
[...]
>> One would be very annoyed if, reading a CSV file, where each of N values
>> on a line correspond to a field of record, if one entry of "?LBC"
>> expanded itself to a dozen entries, screwing everything up.
>>
> Now you're suggesting the _shell_ is going to read and process a CVS
> file???

Be fair: he's giving an analogy.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From cs at zip.com.au  Tue Dec  6 21:48:53 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 7 Dec 2016 13:48:53 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o27q4d$am8$1@dont-email.me>
References: <o27q4d$am8$1@dont-email.me>
Message-ID: <20161207024853.GA19376@cskk.homeip.net>

On 07Dec2016 01:55, BartC <bc at freeuk.com> wrote:
>On 06/12/2016 22:35, Nathan Ernst wrote:
>>One other consideration in regards to globbing in the argument list:
>>there's a static limit to the byte length of argv. On windows, it's 8191
>>bytes
>
>That's the length of the command line, where parameters might still be 
>in the form *.*, ie. pre-splitting and pre-expansion. After converting 
>to argv format (which I believe is done by a C runtime function), that 
>will be in different memory with presumably its own limits.
>
>>(I'm assuming a null-terminator brings that to 8192, which is a weird
>>2**13). For Linux, as of kernal 2.6.25, apparently, the limit is 131072
>>bytes, an equally odd choice of 2**17 bytes.
>
>Maybe they started off at 4KB and 64KB but ran into problems.

I thought Linux had moved on to something more dynamic. Can't remember the 
citation though.

>But even Linux's 128KB will fill if someone wanted a command line that 
>listed 20,000 files individually. But it would be spectacularly bad 
>use of a command line interface which was designed for humans.

One might argue that this is exactly what command lines are to ease (either in 
the shell or in the app): expressing large things easily:

  cat *.txt     # millions of files, anyone?

>>I'm not sure if globs directly to commands bypass these restrictions, but
>>I've seen real world issues with build systems with thousands of files that
>>attempted to pass all dependencies to a target that blew up spectacularly
>>(one of the reasons cl & link on windows accept argument files as input
>>instead of just command line arguments).
>
>So does gcc on Linux (if you mean the @file syntax). It's a tidier way of 
>doing things.

Several things offer ways to read things-expected-to-be-large from files.  
Random example: wget will accept URLs from stdin or a file via its "-i" option, 
instead of always requiring them on the command line.

The typical workaround on UNIX for large argument lists is xargs, though it 
does not fit all use cases.

At some point I reach for (==> meaning "write") a small programme to work 
around such problems. Often a shell script or a python programme.

Cheers,
Cameron Simpson <cs at zip.com.au>


From cs at zip.com.au  Tue Dec  6 21:52:56 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 7 Dec 2016 13:52:56 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o27i21$gqt$1@dont-email.me>
References: <o27i21$gqt$1@dont-email.me>
Message-ID: <20161207025256.GA40439@cskk.homeip.net>

On 06Dec2016 23:37, BartC <bc at freeuk.com> wrote:
>(Actually at this point I haven't got a clue as to how Unix applications are 
>distributed. I guess it's not as simple as just providing a binary executable. 

Given that UNIX systems run on many many different processors, this has limited 
utility :-)

>For the moment, I'm using C source code as every Unix system has a C compiler. 

Feh. Obtainable, generally yes. Provided initially, not always.

>I suppose it could be Python source too, but I doubt if my interpreters 
>written in Python will run quite as briskly -- I find C slow enough for this 
>purpose.)

Depends on the task. C does make for more performant code, but at the expense 
of higher programmer effort. I find Python fits many situations for me. Of 
course not _all_ situations.

The performance trick with interpreted languages (shell, Python) is often to 
offload intensive things to special purpose noninterpreted components, be those 
separate executables or linked in libraries. Use the interpreted portion for 
orchestration and abstraction.

Cheers,
Cameron Simpson <cs at zip.com.au>


From eryksun at gmail.com  Tue Dec  6 21:55:10 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 02:55:10 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <jqoe4c92lu722m4ltlgjeifcr0h8phafom@4ax.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <jqoe4c92lu722m4ltlgjeifcr0h8phafom@4ax.com>
Message-ID: <CACL+1av+8zn7Y8My+t8rE5YAtbgCr0ju-fTa8bveED0=9gtKow@mail.gmail.com>

On Tue, Dec 6, 2016 at 3:05 PM, Random832 <random832 at fastmail.com> wrote:
>
> The fact that Windows was launched at boot by running "win.com" (either
> in autoexec.bat or manually at the command line) created a *perception*
> that windows ran "on top of DOS", but running it really *replaced* DOS
> in memory, putting the CPU into protected mode and everything. The
> ability to boot into (or exit to) DOS was because people still did real
> work (and games) in DOS and the virtual environment of DOS-on-Windows
> didn't perform well enough to be sufficient.

Win16 (and Win32 if it thunked to Win16) called DOS INT 21h functions.
The virtual machine manager implemented some (or most) of these
functions in 32-bit protected mode (VxDs), but it had to maintain DOS
data structures in each VM (e.g. drive working directories; PSPs). It
was like an elaborate 32-bit DOS extender. Of course there was also a
lot of new functionality for Win32 programs such as preemptively
scheduled threads and memory-mapped files.

> I vaguely remember a version of cmd.exe that would run on Windows
> 98 floating around back in the day, but it certainly didn't come with
> the OS. It might have been a pirated Windows NT component.

cmd.exe was ported to Windows 9x, but never distributed with it. On
Windows 9x it lacked Unicode support. Obviously all of the functions
for NTVDM integration were omitted. Its `copy` command called CopyFile
instead of NT's CopyFileEx, so you couldn't Ctrl+C cancel copying a
large file. Reading from the console required custom support for
command-line editing and history -- features that are built into the
NT console. The command prompt would always wait for a child process
to exit, even for GUI apps, because it couldn't call NT's
NtQueryInformationProcess to get the PEB to peek at the subsystem
type.


From steve+python at pearwood.info  Tue Dec  6 21:58:17 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 07 Dec 2016 13:58:17 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
Message-ID: <58477aca$0$1611$c3e8da3$5496439d@news.astraweb.com>

On Wed, 7 Dec 2016 12:25 am, BartC wrote:

> Or someone from Britain visiting the USA and saying OMG, everyone's got
> a gun! Suppose someone runs amok and shoots everybody!

Fun fact: there's a school shooting in the US about once a week.

http://www.huffingtonpost.com.au/entry/school-shootings-since-newtown_n_5480209

The Washington Post does an analysis where they give all sorts of excuses
why shootings on school grounds shouldn't be counted as school shootings.
Here are some of the things that they don't think count as school
shootings:

- an adult couple shooting a cat on school grounds, who told police that
they would have shot children if it had been "god's will";

- a gun that "went off" in a kindergarten student's backpack;

- suicides;

- gang-related shootings;

- accidental discharges;

etc.

https://www.washingtonpost.com/news/fact-checker/wp/2015/06/29/has-there-been-one-school-shooting-per-week-since-sandy-hook/

In other words, if you ignore 90% of school shootings, you can dismiss
concerns of school shootings as a lie.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From orgnut at yahoo.com  Tue Dec  6 22:12:08 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Tue, 6 Dec 2016 19:12:08 -0800
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o266t3$bnd$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me>
Message-ID: <jfKdnakzEpKU49rFnZ2dnUU7-UHNnZ2d@giganews.com>

On 12/06/2016 03:21 AM, BartC wrote:
> On 06/12/2016 07:37, Larry Hudson wrote:
>
>> Now you're suggesting the _shell_ is going to read and process a CVS
>> file???
>
> What follows a shell command is a set of values a,b,c,d,e. What is encountered in a CSV is a set
> of values a,b,c,d,e. You really can't see the similarity?
>
No.  The way they're used is entirely different.

>> I get awfully tired of your constant pontificating about your opinions.
>
> Opinions based on facts: I've given a dozen examples where the shell's auto-expansion can screw
> things up. And I can easily come up with more, as have others in the thread.
>
> People's attitudes seem to be 'So don't that'. Or, 'So what?'.
>
I'll repeat:  They are OPINIONS and "they ain't gonna change nothin', no how, no way, not ever."

MY opinion, which I fully accept "ain't gonna change nothin', no how, no way, not ever" is that 
for someone who claims to be such an expert and experienced programmer, your posts frequently 
show an amazing lack of knowledge.  For example, quoting from another recent post of yours:

 >>    Read the Python documentation for argparse

 > I just tried it, but it was too complex for me to set it up so as to discover with it did 
with * arguments.

Your ignorance is clearly showing.

Finally I have to admit that I'm an old codger and becoming more and more of a curmudgeon.  This 
sometimes shows up in RL, but I generally try to avoid it in public forums like this, but 
occasionally...    :-(

[I suspect that there are others who share my opinion, however, whether that is true or not is 
entirely irrelevant as well.]

That is also all I'm going to say on this subject, so don't bother responding to this post.  I 
won't continue it.

-- 
      -=- Larry -=-


From best_lay at yahoo.com  Tue Dec  6 22:14:24 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 06 Dec 2016 21:14:24 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <o24lul$5r4$1@dont-email.me>
 <QbednUFY9sc8QNjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <87h96iqbor.fsf@elektro.pacujo.net>
 <QbednUBY9sfkdtjFnZ2dnUU7-VOdnZ2d@giganews.com>
 <6100277e-0de8-b376-ddfc-4921461ebb30@gmail.com>
 <mailman.60.1480977569.2253.python-list@python.org>
 <CtOdnQ8Hvtr_adjFnZ2dnUU7-QmdnZ2d@giganews.com>
 <bgchhd-s44.ln1@bernd.nawothnig.dialin.t-online.de>
 <GbmdnUBEet5TedvFnZ2dnUU7-dWdnZ2d@giganews.com>
 <7ef5bad7-4b3d-15db-f4c9-1edd8e5a4013@gmail.com>
 <mailman.90.1481042710.2253.python-list@python.org>
Message-ID: <4budnc9KBJgN49rFnZ2dnUU7-f_NnZ2d@giganews.com>

On Tue, 06 Dec 2016 09:45:05 -0700, Michael Torrie wrote:

> I appreciate your measured response to what could be seen as an
> inflammatory post.

It was inflammatory and I considered a different response but
after the knee jerking, I give it some thought and decided
otherwise.  The simple fact is I'm an outsider here.  Over
the last few months I have received some good advice and help
from some good folks and for that I am grateful.  I do not
want to do anything that could jeopardize that.  So I will
try my best to keep my posts civil.

And I thank you for your words.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Tue Dec  6 22:27:23 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 06 Dec 2016 21:27:23 -0600
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>
 <20161206130635.260c5816@bigbox.christie.dr>
 <mailman.95.1481051221.2253.python-list@python.org>
Message-ID: <5cydnfLMz9UGHNrFnZ2dnUU7-dHNnZ2d@giganews.com>

On Tue, 06 Dec 2016 13:06:35 -0600, Tim Chase wrote:

>

I forgot to mention that I want to include your name in the
final script as a contributor, if that is ok.

You will get a cut of the royalties.  Lets see, how much is
20% of $0.00?  Well, I'll let my account work that out as
soon as she gets home from the grocery.

-- 
<Wildman> GNU/Linux user #557453
The voices in my head may not be real
but they have some good ideas.


From python.list at tim.thechases.com  Tue Dec  6 22:48:22 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Tue, 6 Dec 2016 21:48:22 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <5cydnfLMz9UGHNrFnZ2dnUU7-dHNnZ2d@giganews.com>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <NIqdnfUTavCxYtvFnZ2dnUU7-W_NnZ2d@giganews.com>
 <20161206130635.260c5816@bigbox.christie.dr>
 <mailman.95.1481051221.2253.python-list@python.org>
 <5cydnfLMz9UGHNrFnZ2dnUU7-dHNnZ2d@giganews.com>
Message-ID: <20161206214822.47ef3924@bigbox.christie.dr>

On 2016-12-06 21:27, Wildman via Python-list wrote:
> On Tue, 06 Dec 2016 13:06:35 -0600, Tim Chase wrote:
> I forgot to mention that I want to include your name in the
> final script as a contributor, if that is ok.

No issues here.

> You will get a cut of the royalties.  Lets see, how much is
> 20% of $0.00?

I'm not sure I'd settle for less than 25% of $0. ;-)

-tkc





From eryksun at gmail.com  Tue Dec  6 22:50:27 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 03:50:27 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <18pe4cpr7hons11dpmm4aje7oph1gu73ui@4ax.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <f7ec6625-a1f6-225f-5c18-9700bc8a5484@gmail.com>
 <18pe4cpr7hons11dpmm4aje7oph1gu73ui@4ax.com>
Message-ID: <CACL+1atOi6XkvwqsxiOfs-d7no-ZsPtKT+gbbk1tHbgDFPa=+Q@mail.gmail.com>

On Wed, Dec 7, 2016 at 1:30 AM, Dennis Lee Bieber <wlfraed at ix.netcom.com> wrote:
> (Bah... win10 seems to share parts of the PowerShell console config with
> the old command prompt -- making it nearly impossible [at least in the
> three days I've been on Win10 after the Win7 drive failed] to configure
> PowerShell to look different from the command prompt)

If the console was started from a shortcut, then you're modifying the
shortcut's console settings. If it was started in a new console by
running an executable via Win+R or cmd's `start` command (e.g. `start
powershell`), then the settings are persisted in a subkey of
HKCU\Console. In this case, pinning the icon to the taskbar creates a
new shortcut that uses the default config. You'll have to customize
this pinned shortcut as well.


From eryksun at gmail.com  Tue Dec  6 23:30:01 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 04:30:01 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
Message-ID: <CACL+1atrirrXJ_9sb_EeWoy6dbnrCf4wW07k5xxSwgLTQmyn4g@mail.gmail.com>

On Tue, Dec 6, 2016 at 10:35 PM, Nathan Ernst <nathan.ernst at gmail.com> wrote:
> One other consideration in regards to globbing in the argument list:
> there's a static limit to the byte length of argv. On windows, it's 8191
> bytes (I'm assuming a null-terminator brings that to 8192, which is a weird
> 2**13).

I know strings in cmd.exe are limited to about 8K characters, so
that's probably the limit you observed. The Windows command line can
actually contain up to 65535 bytes. It's a sized UNICODE_STRING, which
uses a C unsigned short for the length. That's 32767 16-bit wide
characters, or 32766 sans the terminating NUL. For example:

parent:

    >>> subprocess.call('python -ic " "' + ' a'*16375 + ' b')

child:

    >>> import sys, win32api
    >>> len(win32api.GetCommandLine())
    32766

    >>> sys.argv[:5]
    ['-c', 'a', 'a', 'a', 'a']
    >>> sys.argv[-1]
    'b'
    >>> len(' '.join(sys.argv)) + len('python i " "')
    32766


From steve+comp.lang.python at pearwood.info  Wed Dec  7 00:15:32 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 07 Dec 2016 16:15:32 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
Message-ID: <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>

On Wednesday 07 December 2016 12:55, BartC wrote:

> But even Linux's 128KB will fill if someone wanted a command line that
> listed 20,000 files individually. But it would be spectacularly bad use
> of a command line interface which was designed for humans.

That's EXACTLY the point of having the shell do globbing.

I want to pass a whole lot of file names to some obscure command. I don't 
remember how that command deals with metacharacters, but it doesn't matter: I 
just say 

    command fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} [a-z]/IMG*

and the shell expands the metacharacters ? {...} * [...] regardless of how much 
smarts the command itself has.

There are thousands of programs I might use, and they may implement who knows 
how many different globbing rules:

- some might support * and ? but not {...}
- some might support SQL-like % and _
- some might insist on regular expressions instead of globs
- some might support the full set of globbing metacharacters, but have bugs
- some might not support any metacharacters
- some might not support ** for recursive wildcards


Don't say "they'll all call a library to do it for them", because even if such 
a library exists, its not necessarily the case that everyone will use it, or 
*want* to use it. There are always those who insist on doing things their own 
way, for their own reasons:

https://www.xkcd.com/1168/

https://explainxkcd.com/wiki/index.php/1168:_tar


I can't easily know in advance what globbing rules I can use for a particular 
command, but I can know what rules my shell uses, and now *all* commands 
support the same, whether they know it or not.




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From marko at pacujo.net  Wed Dec  7 00:30:04 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 07:30:04 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <20161206190043.7fc99068@bigbox.christie.dr>
 <mailman.105.1481072866.2253.python-list@python.org>
Message-ID: <877f7cpaqr.fsf@elektro.pacujo.net>

Tim Chase <python.list at tim.thechases.com>:

> On 2016-12-07 00:29, Marko Rauhamaa wrote:
>> A word a warning: your code doesn't lock /var/run/utmp before
>> access, which is a race condition. The file may be updated at any
>> time, and ordinary file reads may yield corrupted records.
>
> Since the code is reading in record-sized blocks and never writing,
> I'm not sure how much possibility there is for a race condition. At
> worst, I imagine that it would result in reading the old data which
> isn't a horrible condition.

If you read a full record at an offset, you might be right. However,
your code uses Python's buffered I/O:

      with open(utmp_fname, "rb") as f:
          while True:
              bytes = f.read(XTMP_STRUCT_SIZE)

instead of os.open() and os.read().

> For under a certain block-size (PIPE_BUF, which I think used to be
> the minimum POSIX requirement of 512b, but is now 4096b on Linux),
> *nix operating systems were atomic in their reading and writing.

That particular guarantee is true for pipes and FIFOs only.


Marko


From steve+comp.lang.python at pearwood.info  Wed Dec  7 00:57:00 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 07 Dec 2016 16:57 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me> <eaopp7Fj5v8U1@mid.individual.net>
 <o27i21$gqt$1@dont-email.me>
Message-ID: <5847a4ad$0$2758$c3e8da3$76491128@news.astraweb.com>

On Wednesday 07 December 2016 10:37, BartC wrote:

> On 06/12/2016 21:44, Gregory Ewing wrote:
>> BartC wrote:
> 
>>> And the justification? Well, %ENVIRONMENTVARIABLE% gets converted in
>>> Windows, so why not?!
>>
>> No, the justification is that the Unix convention allows
>> the shell to provide certain useful functions that Unix
>> users value.
>>
>> If you don't want those functions, you're free to write
>> your own shell that works however you want. Complaining
>> that everyone *else* should want the same things you
>> want is not reasonable.
> 
> How does that work?
> 
> Suppose I provide an assortment of applications that would work better
> if wildcards are expanded.
> 
> Do I then have to provide one more application, a shell, to be run first
> if someone wants to run any of my applications? Which they then have to
> quit if they want to run some other programs that depend on wildcard
> expansion.

Your question is ambiguous. If we're talking about Greg's tongue-in-cheek 
suggestion that you write your own shell, then the answer is, it depends 
entirely on you. This bartshell that doesn't expand wildcards is your baby, 
doing what *you* want it to do. So its your decision.

But I can tell you that by far the majority of Unix programs don't do their own 
globbing, or even offer it as an opt-in optional feature. Why would they need 
to? So if you want bartshell to be usable on Linux/Unix systems, you'll need 
some sort of opt-in setting to enable wildcard expansion.


> (Actually at this point I haven't got a clue as to how Unix applications
> are distributed. I guess it's not as simple as just providing a binary
> executable. For the moment, I'm using C source code as every Unix system
> has a C compiler. I suppose it could be Python source too, but I doubt
> if my interpreters written in Python will run quite as briskly -- 

bash is about an order of magnitude slower than Python, and its still plenty 
fast enough for practical work.



> I find C slow enough for this purpose.)

http://www.catb.org/jargon/html/story-of-mel.html



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From marko at pacujo.net  Wed Dec  7 02:08:19 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 09:08:19 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>
 <mailman.102.1481065258.2253.python-list@python.org>
Message-ID: <877f7c2p3w.fsf@elektro.pacujo.net>

Michael Torrie <torriem at gmail.com>:

> On 12/06/2016 03:29 PM, Marko Rauhamaa wrote:
>> Another thing is that, as stated before, the runlevel business is
>> legacy. It is still supported by systemd-update-utmp, but for how
>> long is anybody's guess.
>
> System V compatibility is still important to Linux, and as long as it
> is, something resembling a runlevel has to be provided, even it's just
> an abstraction of something else like a systemd target. So any system
> that advertises System V compliance or compatibility will have a
> runlevel.

I wonder if Linux still suffers from this local DoS:

	[ Now no-one else can log in ]

   This is a problem with advisory locking. The fact that anyone can
   create an exclusive lock on a file they can only read! Is this
   behavior appropriate?

   <URL: https://www.redhat.com/archives/linux-security/1996-Novembe
   r/msg00026.html>

Didn't try it, but the utmp API seems hopeless in this regard.


Marko


From nad at python.org  Wed Dec  7 02:26:10 2016
From: nad at python.org (Ned Deily)
Date: Wed, 7 Dec 2016 02:26:10 -0500
Subject: [RELEASE] Python 3.6.0rc1 is now available
Message-ID: <56B63E55-9CD5-4823-8282-449F6E598702@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm excited to announce the availability of Python 3.6.0rc1. 3.6.0rc1
is the release candiate for Python 3.6, the next major release of
Python.

Code for 3.6.0 is now frozen.  Assuming no release critical problems are
found prior to the 3.6.0 final release date, currently 2016-12-16, the
3.6.0 final release will be the same code base as this 3.6.0rc1.
Maintenance releases for the 3.6 series will follow at regular
intervals starting in the first quarter of 2017.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8
* PEP 529 - Change Windows filesystem encoding to UTF-8
* PEP 530 - Asynchronous Comprehensions

Please see "What?s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0rc1 here:

https://www.python.org/downloads/release/python-360rc1/

Note that 3.6.0rc1 is still a preview release and thus its use is not
recommended for production environments.

More information about the release schedule can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []



From steve+comp.lang.python at pearwood.info  Wed Dec  7 02:58:19 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 07 Dec 2016 18:58:19 +1100
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>
 <mailman.102.1481065258.2253.python-list@python.org>
 <877f7c2p3w.fsf@elektro.pacujo.net>
Message-ID: <5847c11c$0$2912$c3e8da3$76491128@news.astraweb.com>

On Wednesday 07 December 2016 18:08, Marko Rauhamaa wrote:

>    This is a problem with advisory locking. The fact that anyone can
>    create an exclusive lock on a file they can only read! Is this
>    behavior appropriate?
> 
>    <URL: https://www.redhat.com/archives/linux-security/1996-Novembe
>    r/msg00026.html>

Whereas if it were mandatory locking, enforced by the OS, it wouldn't be a 
problem?

Here's that URL without the indent and word-wrapping:

https://www.redhat.com/archives/linux-security/1996-November/msg00026.html



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Wed Dec  7 03:30:38 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 07 Dec 2016 19:30:38 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
Message-ID: <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>

On Wednesday 07 December 2016 03:41, BartC wrote:

> On 06/12/2016 15:44, Michael Torrie wrote:
>> On 12/06/2016 04:43 AM, BartC wrote:
> 
>> Yes shell expansion has it's gotchas.  But those can all be learned,
> 
> Yes, learn to avoid wildcards in command parameters at all costs. But we
> both know that is not satisfactory.

Now you're being ridiculous. Can't you at least debate this sensibly?

There are Windows programs which understand globs, like dir. Do you honestly 
expect us to believe that it is okay to use "dir a*" on Windows because dir 
itself expands the glob, but not okay to use "dir a*" on Linux because the 
shell has expanded the glob before dir sees it?

(Yes, there's a Linux dir. It appears to be almost identical to ls.)

 
> And it's not much help when someone types in:
> 
>   program *

Indeed.

Tell me Bart, what's the difference between these?

    # Linux
    rm *

and 

    # Windows
    del *



If you accidentally say "del *" on Windows, say, when you meant "z*" for 
example, and Windows deletes all the files in the current directory, do you say 
to yourself...?

   "Thank the deities that * was expanded by the del command rather 
   than the shell! Imagine how awful it would have been if the shell
   did the expansion!"


> and the program has to try and clean up the mess, if it can see it is a
> mess. But remember:
> 
>    cp *.c
> 
> There might be some irate users out there if it can't detect a simple
> user error like that.

*shrug* You can blow your foot off with any sufficiently powerful set of tools, 
if you're not paying attention.

I see that the Windows version of copy avoids this *by accident*:

C:\Documents and Settings\Steve>dir a*
 Volume in drive C has no label.
 Volume Serial Number is 9CAB-3B4A

 Directory of C:\Documents and Settings\Steve

07/12/2016  06:05 PM                 4 a1
07/12/2016  06:05 PM                 4 a2
               2 File(s)              8 bytes
               0 Dir(s)   1,672,613,888 bytes free

C:\Documents and Settings\Steve>copy a*
a1
The file cannot be copied onto itself.
        0 file(s) copied.



Instead of accidentally copying a1 over a2, it accidentally tries to copy a1 
over a1, and fails. At least, that's what it looks like to me. Any Windows 
expert like to chime in?


The bottom line is, it doesn't matter where the wildcards are expanded. Whether 
the shell does it, or the application itself, they're dangerous by their very 
nature. You're asking the computer to operate on a set of files you specify 
using fuzzy matching. It is easy to match too much, or too little, if you 
aren't careful.


>> whereas it's much harder to learn and remember all the gotchas and bugs
>> of many individual applications' unique ways of dealing with globs. I'd
>> rather deal with shells.
>>
> 
> OK, I understand. Having a program launcher indiscriminately transform
> input A into A' is /much/ better than having it done by the program,

For the use-cases that guided the design and evolution of Unix, ABSOLUTELY.

That doesn't mean that the same will apply to all uses of the computer command 
line. But it does apply to the great majority of uses by sys admins on Unix 
systems. They designed the system to work the way they want.

I get that its not the way that you like, but fortunately for you, they 
provided not one, but TWO easy ways to get the results you want:

- escaping the metacharacters you want to be left untouched;

- disabling globbing (for either a single command, a session, or permanently).


If *your* design was paramount, it would be up to each and every application to 
choose whether or not to support globbing and environment variable expansion, 
and hard luck if the application I needed to use didn't understand the globs I 
wanted.


> even if the program doesn't want it and it is not meaningful.

*shrug* That's the cost you pay. It's not a big cost, since it costs as little 
as a single character (backslash) to escape something, but it is a cost.


> The fact that you also lose format information (is it three parameters,
> or one parameter transformed into three) is an extra bonus.

Here's a two-line Python program to prove what nonsense that is.

steve at runes:~$ cat argv.py 
import sys
print( sys.argv )


And in use:

steve at runes:~$ python argv.py a\* z\*
['argv.py', 'a*', 'z*']



> This is clearly much better than any other scheme because:
> 
> (1) It's Unix not Windows; everything in Unix is always better and
> always make sense.

[snarky, sarcastic answer to match the tone of your post]

Everything except the tar command.


> (2) There have been 40 years of it working this way and there have never
> been any problems. (That is, 40 years of writing programs with
> stultified command line interfaces to make sure that is always the case.
> [As for 'problems' such as the 'cp *.c' one, that's a feature!]

Don't be absurd.

Any Unix sys admin can tell you stories of accidentally globbing too much and 
screwing something up. And any Windows sys admin can tell you the same stories, 
even though globbing is handled by the individual programs rather than the 
shell.

(At least, any Windows sys admin who uses the command line, or writes batch 
files. I dare say those who use nothing but GUI tools have their own, 
different, horror stories.)

But that's the nature of the job, not the choice of where the globs are 
expanded. Certainly Unix sys admins will cheerfully admit that bash and other 
shells are designed for quick, interactive use, they aren't ideal for writing 
large, complex, maintainable systems.

But Unix systems are the end product of 40 years of directed evolution by 
thousands of sys admins who have designed the shells to work the way they want.

And it looks like Windows is ever-so-slowly converging on the same, or at least 
similar, sets of conventions: by copying Unix globbing metacharacters in the 
first place, then by adding them to the C library to make it easy for 
applications to do their own globbing, then by moving to PowerShell, and the 
latest step is Microsoft's partnership with Canonical (the company behind 
Ubuntu Linux) to bring bash to Windows.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From __peter__ at web.de  Wed Dec  7 03:50:07 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 07 Dec 2016 09:50:07 +0100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
Message-ID: <o28ifv$ros$1@blaine.gmane.org>

Steven D'Aprano wrote:

> Tell me Bart, what's the difference between these?
> 
> # Linux
> rm *
> 
> and
> 
> # Windows
> del *

Is there an equivalent to

# touch -- -r 

on Windows? Like in

$ tree
.
|-- -r
|-- a
|   `-- b
|       `-- c
|           `-- bar.txt
`-- foo.txt

3 directories, 3 files
$ rm *
$ tree
.
`-- -r

0 directories, 1 file
$ 




From dr.roman.graf at gmail.com  Wed Dec  7 04:08:03 2016
From: dr.roman.graf at gmail.com (dr.roman.graf at gmail.com)
Date: Wed, 7 Dec 2016 01:08:03 -0800 (PST)
Subject: Django broken pipe error
In-Reply-To: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
References: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
Message-ID: <843bd5a8-f127-4fdc-8817-0b1645c1872d@googlegroups.com>

On Tuesday, December 6, 2016 at 3:22:13 PM UTC+1, dr.rom... at gmail.com wrote:
> Hi,
> 
> I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that apparently is a not fixed Django bug. Does anybody now how to fix it? I've been searching a lot and didn't find any solution.
> 
> This error happens very irregularly by Post request in Django. Sometimes it works sometimes not: 
> 
> [06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 129, in __init__
>     super(WSGIRequestHandler, self).__init__(*args, **kwargs)
>   File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
>     self.finish()
>   File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
>     self.wfile.close()
>   File "/usr/lib/python2.7/socket.py", line 279, in close
>     self.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
>     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> ----------------------------------------
> Exception happened during processing of request from ('127.0.0.1', 38224)
> 
> It is also described in: https://www.reddit.com/r/joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/
> 
> On https://bugs.python.org/issue14574 is stated that this error should already be fixed but apparently not.
> 
> Best regards,
> 
> Roman

Thank you Justin,

I'm on the dev server and should present results in this way. 

Yes, I use manage.py runserver --insecure to start the server (from PyCharm).

My views.py call:

@detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
def export_report(request):
    body_unicode = request.body.decode('utf-8')
    body_str = body_unicode.encode('ascii','ignore')
    attr_list = body_str.split('&')
    attr_dict = {}
    if (len(attr_list) > 0):
        for attr in attr_list:
           ...
           key = key_value_pair[0]
           attr_dict[key] = key_value_pair[1]
    trend = trends.calculate_trend(
        attr_dict['search_phrase']
        , attr_dict['time_from']
        , attr_dict['time_to']
        , attr_dict['time_scale']
    )
    attr_dict['trend'] = trend
    res = str(json.dumps(attr_dict))
    return HttpResponse(res, content_type="text/plain; charset=utf-8")

and trend calculation in trends.py with database calls:

def calculate_trend(query_phrase, time_from, time_to, time_scale):
    # check in database if trend already exists
    try:
        db_trend = Trend.objects.get(pk=query_phrase)
        if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
                and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
                and db_trend.granularity == time_scale:
            logger.info("trend already exists.")
            existing_trend_dict = ast.literal_eval(db_trend.content)
            return existing_trend_dict
    except Trend.DoesNotExist:
        logger.info("It is a new trend search.")
    trend_dict = {}
    start_time = pd.Timestamp(value[0])
    end_time = pd.Timestamp(value[-1])
    freq = ... get frequency using pandas lib
    trend_dict[key] = freq
    json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
    trend = Trend(
        phrase=query_phrase,
        content=json_trend_content,
        from_time=time_from,
        to_time=time_to,
        granularity=time_scale,
    )
    if trend is not None:
        try:
            db_trend = Trend.objects.get(pk=query_phrase)
            db_trend.delete()
            logger.info("delete old trend: %s. " % trend)
        except Trend.DoesNotExist:
            logger.info("create trend: %s. " % trend)
        trend.save()
    return trend_dict

Thank you in advance!

Roman


From rosuav at gmail.com  Wed Dec  7 04:43:39 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 7 Dec 2016 20:43:39 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
Message-ID: <CAPTjJmomJk0-FDEeHvEApfs1MEVX6rax0LP_9rHm3xJP7epFjQ@mail.gmail.com>

On Wed, Dec 7, 2016 at 7:30 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> If you accidentally say "del *" on Windows, say, when you meant "z*" for
> example, and Windows deletes all the files in the current directory, do you say
> to yourself...?
>
>    "Thank the deities that * was expanded by the del command rather
>    than the shell! Imagine how awful it would have been if the shell
>    did the expansion!"

There is one special case: if you use "*.*" on Windows, the del
command will prompt. However, there are plenty of other examples where
this special case doesn't apply.

> I see that the Windows version of copy avoids this *by accident*:
>
> C:\Documents and Settings\Steve>copy a*
> a1
> The file cannot be copied onto itself.
>         0 file(s) copied.
>
>
>
> Instead of accidentally copying a1 over a2, it accidentally tries to copy a1
> over a1, and fails. At least, that's what it looks like to me. Any Windows
> expert like to chime in?

Not a Windows expert, but I happen to know this one; and it's another
special case. If you provide only one argument (and a glob does count
as a single argument), copy assumes you mean "to the current
directory" - ie an implicit "." as a second argument.

So you've landed on two special cases in Windows commands that take
advantage of the fact that the app has to expand globs. They give a
marginally better result at the cost of consistency. I believe the Zen
of Python has something to say about special cases; while there
definitely are times when that line is wrong, there are a lot of times
when it's right, and forfeiting consistency is usually too high a cost
for the benefit gained.

>> (2) There have been 40 years of it working this way and there have never
>> been any problems. (That is, 40 years of writing programs with
>> stultified command line interfaces to make sure that is always the case.
>> [As for 'problems' such as the 'cp *.c' one, that's a feature!]
>
> Don't be absurd.
>
> Any Unix sys admin can tell you stories of accidentally globbing too much and
> screwing something up. And any Windows sys admin can tell you the same stories,
> even though globbing is handled by the individual programs rather than the
> shell.

And a lot of the over-globbing examples depend on very specific
situations. With "cp *.c" and no destination, Unix cp will silently do
the wrong thing ONLY if there are exactly two files matching that
glob, or if everything matching that glob is a file except the last,
which is a directory. (With most globs, that's unlikely.) With Windows
"copy *.c", it gives an obscure error message ONLY because you didn't
have a path on the glob; if you say "copy source\*.c", it'll silently
do the wrong thing.

Let's face it. You should be using git anyway. Then you have a
protection *no matter what* you mess up - just check out any files you
messed up. Tada!

> (At least, any Windows sys admin who uses the command line, or writes batch
> files. I dare say those who use nothing but GUI tools have their own,
> different, horror stories.)

Oh, absolutely. How about a Windows domain controller that BSODed
multiple times just trying to pull up the config program?

http://thedailywtf.com/articles/the-infrastructure

Part of that WTF is that the domain controller had no remote access.
But what's the cost (in server load) of a GUI-based remote admin
facility? Decidedly non-zero. How about good ol' SSH? So light that
I've been able to use it to remote in to a computer that had gotten
itself into some sort of infinite loop, and find out what was going
wrong. And fix it without rebooting, too. If your solution to
everything is "bah, just reboot", then you don't have a solution.

> But Unix systems are the end product of 40 years of directed evolution by
> thousands of sys admins who have designed the shells to work the way they want.
>
> And it looks like Windows is ever-so-slowly converging on the same, or at least
> similar, sets of conventions: by copying Unix globbing metacharacters in the
> first place, then by adding them to the C library to make it easy for
> applications to do their own globbing, then by moving to PowerShell, and the
> latest step is Microsoft's partnership with Canonical (the company behind
> Ubuntu Linux) to bring bash to Windows.

Exactly. Back in the 1990s, we had the beginnings of Windows NT, which
was designed for servers. It had (if I recall correctly) no concept of
file/directory permissions, little or no notion of process privilege,
and definitely no way to restrict one process to X amount of memory
and Y amount of processing. Since then, Windows has progressively
added features that Unix has had for decades, because those features
are critical. Those who don't understand Unix are forced to reinvent
it, piece by piece, badly, and then finally rip it out and start
over...

ChrisA


From marko at pacujo.net  Wed Dec  7 05:10:23 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 12:10:23 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>
 <mailman.102.1481065258.2253.python-list@python.org>
 <877f7c2p3w.fsf@elektro.pacujo.net>
 <5847c11c$0$2912$c3e8da3$76491128@news.astraweb.com>
Message-ID: <871sxk2gog.fsf@elektro.pacujo.net>

Steven D'Aprano <steve+comp.lang.python at pearwood.info>:

> Whereas if it were mandatory locking, enforced by the OS, it wouldn't
> be a problem?

The point is, the utmp scheme seems to be fundamentally broken. You
can't use a regular file for this kind of communication.


Marko


From bc at freeuk.com  Wed Dec  7 06:54:35 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 11:54:35 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
Message-ID: <o28t7e$2s9$1@dont-email.me>

On 07/12/2016 08:30, Steven D'Aprano wrote:
> On Wednesday 07 December 2016 03:41, BartC wrote:
>
>> On 06/12/2016 15:44, Michael Torrie wrote:
>>> On 12/06/2016 04:43 AM, BartC wrote:
>>
>>> Yes shell expansion has it's gotchas.  But those can all be learned,
>>
>> Yes, learn to avoid wildcards in command parameters at all costs. But we
>> both know that is not satisfactory.
>
> Now you're being ridiculous. Can't you at least debate this sensibly?
>
> There are Windows programs which understand globs, like dir. Do you honestly
> expect us to believe that it is okay to use "dir a*" on Windows because dir
> itself expands the glob, but not okay to use "dir a*" on Linux because the
> shell has expanded the glob before dir sees it?

You don't get it.

The problem is the program not seeing the original unexpanded parameters 
that it wants, with extra information that is lost when indiscriminately 
expanded.

> Tell me Bart, what's the difference between these?
>
>     # Linux
>     rm *
>
> and
>
>     # Windows
>     del *

It asks 'Are your sure?'. But that is beside the point. And commands 
such as 'del' /are/ crude.

But if a program existed that took N filename parameters with the 
purpose of deleting each of them, then it can't tell if they were typed 
in individually (so indicating a stronger intent), or whether a finger 
slipped on a single filename and a "*" was added, in which case it could 
choose to ask for confirmation *if* it saw that this is a wildcard).

'Globbing' (if that means auto expansion) is taking away choice.

> I see that the Windows version of copy avoids this *by accident*:

> C:\Documents and Settings\Steve>copy a*
> a1
> The file cannot be copied onto itself.
>         0 file(s) copied.

No, even if it did copy the file, it would be harmless. 'copy' chose to 
make the second parameter option, and to make it default to "."

> Instead of accidentally copying a1 over a2, it accidentally tries to copy a1
> over a1, and fails. At least, that's what it looks like to me. Any Windows
> expert like to chime in?

a1 would be copied to a1, a2 to a2. That's how it has to work otherwise 
all files would be renamed when copied elsewhere!

>
> The bottom line is, it doesn't matter where the wildcards are expanded. Whether
> the shell does it, or the application itself, they're dangerous by their very
> nature.

They can be dangerous with certain commands and programs that accept 
wildcards (which aren't many actually; I don't think any of my programs do).

With automatic expansion, then EVERY program can become dangerous. Can 
no one else see this?

> I get that its not the way that you like, but fortunately for you, they
> provided not one, but TWO easy ways to get the results you want:
>
> - escaping the metacharacters you want to be left untouched;
>
> - disabling globbing (for either a single command, a session, or permanently).

If the default is to have it on, then that's not satisfactory.

> If *your* design was paramount, it would be up to each and every application to
> choose whether or not to support globbing and environment variable expansion,
> and hard luck if the application I needed to use didn't understand the globs I
> wanted.

How many programs take parameters that specify multiple files and work 
on them together? As I said, I think hardly any of mine do.

>> The fact that you also lose format information (is it three parameters,
>> or one parameter transformed into three) is an extra bonus.
>
> Here's a two-line Python program to prove what nonsense that is.
>
> steve at runes:~$ cat argv.py
> import sys
> print( sys.argv )
>
>
> And in use:
>
> steve at runes:~$ python argv.py a\* z\*
> ['argv.py', 'a*', 'z*']

But in real use it will be:

  python argv.py a* z*

It took me a while to recognise the significance of the "\". In Windows 
that's a path separator.

Yes, there are any number of ways that *you* or *I* can get AROUND the 
problem. But my background in the past has been with users (and talking 
through bunches of console commands on the phone).

But the problem is still there.

> And it looks like Windows is ever-so-slowly converging on the same, or at least
> similar, sets of conventions: by copying Unix globbing metacharacters in the

(I'm sure I used these in the 1970s. Or was DEC copying Unix?)

> first place, then by adding them to the C library to make it easy for
> applications to do their own globbing, then by moving to PowerShell, and the
> latest step is Microsoft's partnership with Canonical (the company behind
> Ubuntu Linux) to bring bash to Windows.

Most open source software seems to be written in C and seems to require 
executing a 20,000 or 30,000-line bash 'configure' script, itself 
dependent on a dozen Unix utilities, before it's possible to build it. 
Even if the resulting product is cross-platform.

So that's not surprising, if everyone is obliged to install Msys or 
Cygwin anyway. (Apart from me, as I refuse to do anything like that. If 
a program is in C, then I should need only a C compiler. Any software I 
make available takes exactly that approach.)

-- 
Bartc


From bc at freeuk.com  Wed Dec  7 07:23:00 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 12:23:00 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o28usn$89t$1@dont-email.me>

On 07/12/2016 05:15, Steven D'Aprano wrote:
> On Wednesday 07 December 2016 12:55, BartC wrote:
>
>> But even Linux's 128KB will fill if someone wanted a command line that
>> listed 20,000 files individually. But it would be spectacularly bad use
>> of a command line interface which was designed for humans.
>
> That's EXACTLY the point of having the shell do globbing.

Sorry, it's got NOTHING to do with it!

Here's a command that works on all one million files in the current path:

  program *.*

11 characters, so Linux still has 131061 left (assuming 'program ' is 
part of it).

But Windows will also have 8181 characters left (or 8180, whatever). And 
the shell hasn't needed to do any 'globbing'.

Now we come to the next bit, where the program in question has an entry 
point that looks like 'main(argc, argv)', where some function splits up 
the command line.

In Windows, argc is 2 (name of program, and the "*.*" parameter). If the 
program wants to process all one million files, it will have to scan 
them one at a time, but it does not need to build a list first.

In Linx, argc will be 1000001, and the program will still need scan them 
one at a time (although it won't need to call an OS function to do the 
iteration).

In neither case was the command line limit an issue.

But I can see where you're coming from: if someone was silly enough to 
want to specify all one million filenames on the command line, then they 
will eventually come up to the limit. On both systems.

That is not going to happen however. And is nothing to do with how or 
where *.* expansion is done.


>     command fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} [a-z]/IMG*


   list fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} \
     [a-z]/IMG* > files

   command @files

Only 'list' needs to understand all this crap (and more besides). 
Applications only need to understand '@'. Or maybe @files /can/ be 
expanded by the shell, more acceptable here because it's explicit.

-- 
Bartc


From rosuav at gmail.com  Wed Dec  7 07:39:55 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 7 Dec 2016 23:39:55 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o28t7e$2s9$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
Message-ID: <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>

On Wed, Dec 7, 2016 at 10:54 PM, BartC <bc at freeuk.com> wrote:
> But if a program existed that took N filename parameters with the purpose of
> deleting each of them, then it can't tell if they were typed in individually
> (so indicating a stronger intent), or whether a finger slipped on a single
> filename and a "*" was added, in which case it could choose to ask for
> confirmation *if* it saw that this is a wildcard).
>
> 'Globbing' (if that means auto expansion) is taking away choice.

And app globbing takes away choice, too. How would you, with the
Windows 'del' command, remove a single file that has a question mark
in the name? Or are you depending entirely on the fact that Windows
doesn't let you do that? Here's some of my music collection:

$ find Music/ -name *\\?*
Music/BeautyStone2/25: Act III. Change of scene?.ogg
Music/BeautyStone2/08: Act II, Scene 1. No. 13 continued. Philip:"How
say you?".ogg
Music/Seekers/CD2 - 1964-1965/17. What Have They Done To The Rain?.mp3
Music/Seekers/CD4 - Hits. B-sides & 90s/14. When Will The Good Apples Fall?.mp3
Music/BeautyStone1/06: Act I, Scene 1. No. 4. Quartet. Simon:"Who
stands within?".ogg
Music/G&S/Pirates of Penzance/111 What ought we to do, Gentle sisters, say?.mp3
Music/G&S/Pirates of Penzance/MusicOnly/111 What ought we to do,
Gentle sisters, say?.ogg

Note that two of the Beauty Stone tracks include quotes as well as
question marks. How do you identify those? Let's say you want to play
one of these in VLC, and then maybe you decide that the track in
Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild
it from the one in the parent directory. How does that work on
Windows? If you say "it doesn't", then (a) you have taken away choice
on a fundamental level, (b) you have your head in the sand, and (c)
you still haven't solved the problem of percent signs, carets, and so
on, which are perfectly legal in file names, but have meaning to the
shell.

> They can be dangerous with certain commands and programs that accept
> wildcards (which aren't many actually; I don't think any of my programs do).

And that right there is a mark against the Windows model. Your program
has to consciously accept wildcards. Mine (on Unix) just has to accept
file names, and *everything will work*.

> If the default is to have it on, then that's not satisfactory.

One line in your shell initialization script will change that default.

>> If *your* design was paramount, it would be up to each and every
>> application to
>> choose whether or not to support globbing and environment variable
>> expansion,
>> and hard luck if the application I needed to use didn't understand the
>> globs I
>> wanted.
>
>
> How many programs take parameters that specify multiple files and work on
> them together? As I said, I think hardly any of mine do.

Once again, you assume that because YOU PERSONALLY write no programs
that need this feature, the feature ITSELF is a bad thing. Can you not
see the arrogance of that?

> It took me a while to recognise the significance of the "\". In Windows
> that's a path separator.

You call yourself a Python programmer and are surprised to find that
backslash is the escape character? It's a pretty common convention.

> Most open source software seems to be written in C and seems to require
> executing a 20,000 or 30,000-line bash 'configure' script, itself dependent
> on a dozen Unix utilities, before it's possible to build it. Even if the
> resulting product is cross-platform.

Not everything is like that. Here, find the configure scripts in these
projects of mine:

https://github.com/Rosuav/Gypsum
https://github.com/Rosuav/StilleBot
https://github.com/Rosuav/FrozenOST
https://github.com/Rosuav/LetMeKnow
https://github.com/Rosuav/zawinski
https://github.com/Rosuav/full-stack-app
https://github.com/Rosuav/miniature-succotash
https://github.com/Rosuav/Yosemite
https://github.com/Rosuav/Hymns
https://github.com/Rosuav/pike-alsa

I think the longest config file is the Hymns collection, with a
twenty-line makefile. Don't judge all open source on what you see in
Mozilla Thunderbird. :)

ChrisA


From eryksun at gmail.com  Wed Dec  7 08:07:17 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 13:07:17 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
Message-ID: <CACL+1asATK8ZgXFwOGFYAR9OqxEy_xMxdWdqUbssqnL+18e+Vg@mail.gmail.com>

On Wed, Dec 7, 2016 at 8:30 AM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
>
> There are Windows programs which understand globs, like dir. Do you honestly
> expect us to believe that it is okay to use "dir a*" on Windows because dir
> itself expands the glob, but not okay to use "dir a*" on Linux because the
> shell has expanded the glob before dir sees it?

"dir" is a built-in shell command. But yes, there are programs that
understand globbing. A C/C++ program can link with wsetargv.obj to get
this feature. That said, I hadn't actually ever used this feature
prior to reading this thread. I linked it into a custom of python.exe,
and I'm not happy with the implementation because it doesn't allow
escaping wildcards (confirmed in the source distributed with VS 2015).

> latest step is Microsoft's partnership with Canonical (the company behind
> Ubuntu Linux) to bring bash to Windows.

The partnership with Canonical is to provide an entire Linux
distribution, sans the kernel. The Windows Subsystem for Linux (WSL)
works by translating Linux system calls to the NT kernel. It directly
executes unmodified ELF binaries. I think it was a mistake to call it
"bash on Windows". It seems like people think it's just a Windows port
of bash.


From marko at pacujo.net  Wed Dec  7 08:21:47 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 15:21:47 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
 <mailman.119.1481114405.2253.python-list@python.org>
Message-ID: <87wpfb27tg.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> $ find Music/ -name *\\?*

I do recommend quoting your wildcard characters.

Yet further nasty security pitfalls in bash programming:

 5. Wildcards that don't match any file are left unexpanded.

 6. Commands behave unexpectedly when given null lists:

    $ ls a b c
    a     b     c
    $ ls a b
    a     b
    $ ls a
    a
    $ ls
    a     b     c     d


Marko


From steve+python at pearwood.info  Wed Dec  7 08:26:44 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 08 Dec 2016 00:26:44 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <CACL+1asATK8ZgXFwOGFYAR9OqxEy_xMxdWdqUbssqnL+18e+Vg@mail.gmail.com>
 <mailman.120.1481116081.2253.python-list@python.org>
Message-ID: <58480e16$0$1589$c3e8da3$5496439d@news.astraweb.com>

On Thu, 8 Dec 2016 12:07 am, eryk sun wrote:

>> latest step is Microsoft's partnership with Canonical (the company behind
>> Ubuntu Linux) to bring bash to Windows.
> 
> The partnership with Canonical is to provide an entire Linux
> distribution, sans the kernel. The Windows Subsystem for Linux (WSL)
> works by translating Linux system calls to the NT kernel. It directly
> executes unmodified ELF binaries. I think it was a mistake to call it
> "bash on Windows". It seems like people think it's just a Windows port
> of bash.


/slaps forehead


Of course it is! Yes, you're right -- its more than just bash. I actually
knew that, but forgot, so thanks for the reminder.

The reason I misremembered it as just bash is when I learned about this,
after the initial chorus of people saying "hang on, its a day early for
April's Fool", the majority of comments I saw were from dual Windows/Linux
system administrators saying "At last I can use bash on Windows!".




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From eryksun at gmail.com  Wed Dec  7 08:51:57 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 13:51:57 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPTjJmomJk0-FDEeHvEApfs1MEVX6rax0LP_9rHm3xJP7epFjQ@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmomJk0-FDEeHvEApfs1MEVX6rax0LP_9rHm3xJP7epFjQ@mail.gmail.com>
Message-ID: <CACL+1avW=yLNBEaDWDeepF5pEdy4Sgx+=5Kpe-B6b_zHRiNNjg@mail.gmail.com>

On Wed, Dec 7, 2016 at 9:43 AM, Chris Angelico <rosuav at gmail.com> wrote:
> Exactly. Back in the 1990s, we had the beginnings of Windows NT, which
> was designed for servers. It had (if I recall correctly) no concept of
> file/directory permissions, little or no notion of process privilege,
> and definitely no way to restrict one process to X amount of memory
> and Y amount of processing.

Do you have a name or a link for this system that pre-dates NT? At the
time I thought Microsoft only had MS-DOS and OS/2 (and maybe still
Xenix?). I know they hired Dave Cutler (the designer of VMS) and
several other DEC programmers in late 1988 to develop NT OS/2. It took
them 5 years, and along the way the design switched from NT OS/2 to
Windows NT (DOS-based Windows had surged in popularity with the
release of Windows 3.x, and the business relationship with IBM fell
apart). They relegated the OS/2 and POSIX subsystems to the console
only, without GUI support (similar to the new Linux subsystem).

> Since then, Windows has progressively
> added features that Unix has had for decades, because those features
> are critical. Those who don't understand Unix are forced to reinvent
> it, piece by piece, badly, and then finally rip it out and start
> over...

NT was designed to run a Unix subsystem and meet U.S. DoD security
requirements, so it's naturally going to share a lot of capabilities
with Unix. But the similarities are superficial; the design decisions
are fundamentally different in many ways. (The differences are more
obvious in native NT and kernel development.) No one could say the NT
design team was copying Unix. OTOH, how much they pilfered from their
previous work on VMS is debatable. IIRC, Microsoft ultimately reached
a settlement out of court with DEC on this issue, which was the main
reason NT was ported to the DEC Alpha processor in the 90s.


From grant.b.edwards at gmail.com  Wed Dec  7 09:22:44 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Wed, 7 Dec 2016 14:22:44 +0000 (UTC)
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>
 <mailman.102.1481065258.2253.python-list@python.org>
 <877f7c2p3w.fsf@elektro.pacujo.net>
 <5847c11c$0$2912$c3e8da3$76491128@news.astraweb.com>
 <871sxk2gog.fsf@elektro.pacujo.net>
Message-ID: <o295vk$qbe$1@blaine.gmane.org>

On 2016-12-07, Marko Rauhamaa <marko at pacujo.net> wrote:
> Steven D'Aprano <steve+comp.lang.python at pearwood.info>:
>
>> Whereas if it were mandatory locking, enforced by the OS, it
>> wouldn't be a problem?
>
> The point is, the utmp scheme seems to be fundamentally broken. You
> can't use a regular file for this kind of communication.

There are a few things in Unix that are fundamentally broken and
really just can't be used for the things they are intended for (serial
ports come to mind).

However, that doesn't seem to prevent them from having been used
sucessfully that way way for 40 years. ;)

-- 
Grant Edwards               grant.b.edwards        Yow! I KAISER ROLL?!
                                  at               What good is a Kaiser Roll
                              gmail.com            without a little COLE SLAW
                                                   on the SIDE?



From steve+python at pearwood.info  Wed Dec  7 09:34:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 08 Dec 2016 01:34:07 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
Message-ID: <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Wed, 7 Dec 2016 11:23 pm, BartC wrote:

> On 07/12/2016 05:15, Steven D'Aprano wrote:
>> On Wednesday 07 December 2016 12:55, BartC wrote:
>>
>>> But even Linux's 128KB will fill if someone wanted a command line that
>>> listed 20,000 files individually. But it would be spectacularly bad use
>>> of a command line interface which was designed for humans.
>>
>> That's EXACTLY the point of having the shell do globbing.
> 
> Sorry, it's got NOTHING to do with it!
> 
> Here's a command that works on all one million files in the current path:
> 
>   program *.*


No, sorry, you're wrong. That specific program is buggy, and its
implementation of globbing is unreliable. I think you meant this one:

    prog *.*

except, no, that one uses regular expressions, because the author is an
opinionated PITA who dislikes globs. So you would have to write:

    prog .*?\..*

but that's okay, you remembered that, didn't you? I hope so, otherwise it
will BSOD if you call it with an invalid regex.

Perhaps you meant to say

   appl *.*

but only version 4.7 or higher, it didn't get support for globbing before
then.

(If you're going to make up imaginary programs that behave the way you
expect, I'm allowed to do the same.)


> 11 characters, so Linux still has 131061 left (assuming 'program ' is
> part of it).

I don't know why you are so hung up over the number of characters here, or
this bogeyman of "one million files" in a directory. Who does such a thing?
Most sys admins will go their entire career without ever needing to process
a million files in a single directory, or even split over a dozen
directories.

But for those who do need to do something like that, there are ways to deal
with it. Nobody (except in your imagination) says that the Unix way of
doing things is optimal for all possible scenarios. As I keep saying, and
you keep ignoring, the Unix way is the product of forty years of directed
evolution by people who have chosen this way because it works best for 90,
95% of what they do. Of course there are exceptions.

I understand that *your* needs are different. I've acknowledged that
repeatedly. And I've repeatedly told you how, for the sake of just a few
extra keystrokes, you can get the results you want on Linux:

- use backslashes to escape individual metacharacters;
- quote the arguments;

(but beware that "x" and 'x' are different in bash, and I'm afraid I'm not
expert enough to remember what the differences are. I just always use "x"
and hope :-)

- disable globbing, using "set -o noglob" in bash.

We get it that your needs are not those of the typical Unix sys admin. Unix
command line tools are designed for Unix sys admins, not you. But they're
not idiots, despite what you apparently think, and provide a way to bail
out of the normal pre-processing when needed.


[...]
> In Windows, argc is 2 (name of program, and the "*.*" parameter). If the
> program wants to process all one million files, it will have to scan
> them one at a time, but it does not need to build a list first.

If you want to pass a file spec to a Unix program, all you need do is quote
the file spec:

ls "*.*"

But it won't do you any good. I don't know any Unix programs that provide
file spec processing. (That's not to say that there are absolutely none,
only that I don't know of any.) That's simply not considered a necessary or
useful feature. But if you want to provide that feature in your own
programs, go right ahead.

But... how does your program distinguish between the file spec *.* which
should be expanded, and the literal file *.* which shouldn't be expanded?

You need an escape mechanism too. You've done that, of course. Right?


[...]
>>     command fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg}
>>     [a-z]/IMG*
> 
> 
>    list fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} \
>      [a-z]/IMG* > files
> 
>    command @files

*Requiring* this would be completely unacceptable. Forcing the user to write
two lines (one to create a variable, then to use it) every time they needed
a glob expansion would go down among sys admins like a lead balloon.

But for those times when you really need to repeat a complex set of
arguments, bash has you covered:

[steve at ando ~]$ export spec="a* z*"
[steve at ando ~]$ echo "$spec"
a* z*
[steve at ando ~]$ ls $spec
aaa-bbb.odt  addresses        ararat        arpaio.txt      a?         zip




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From p.f.moore at gmail.com  Wed Dec  7 09:54:44 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 7 Dec 2016 06:54:44 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.88.1481042038.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me>
 <1e112d8d-d8bf-426f-8b0f-d4e8e82db23d@gmail.com>
 <mailman.88.1481042038.2253.python-list@python.org>
Message-ID: <8275270e-6e9d-44e0-b2b0-a6b30cc21142@googlegroups.com>

On Tuesday, 6 December 2016 16:34:10 UTC, Michael Torrie  wrote:
> On 12/06/2016 06:52 AM, BartC wrote:
> > Then you don't get utterly ridiculous and dangerous behaviour such as 
> > the cp example Paul Moore came up with (that trumps most of mine actually):
> 
> It's potentially dangerous agreed.

Note, by the way, that BartC is misquoting me - I never said that behaviour was "ridiculous and dangerous". I pointed out that it was potentially dangerous and needs to be learned and understood. And that it's the result of a trade-off made by Unix, where Windows makes a different trade-off.

My point was in fact that BartC's position, that expecting completely different environments (Unix shells vs Windows shells vs Popen) to behave identically, is silly. *All* environments make usability trade-offs, and do so in different ways. Users need to learn these, or accept that any mistakes they make are due to their own lack of understanding, and not because, as BartC repeatedly (and rather offensively) asserts, the environments they are not familiar with are "stupid".

Paul


From bc at freeuk.com  Wed Dec  7 09:57:23 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 14:57:23 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.119.1481114405.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
 <mailman.119.1481114405.2253.python-list@python.org>
Message-ID: <o297u6$7u0$1@dont-email.me>

On 07/12/2016 12:39, Chris Angelico wrote:
> On Wed, Dec 7, 2016 at 10:54 PM, BartC <bc at freeuk.com> wrote:
>> But if a program existed that took N filename parameters with the purpose of
>> deleting each of them, then it can't tell if they were typed in individually
>> (so indicating a stronger intent), or whether a finger slipped on a single
>> filename and a "*" was added, in which case it could choose to ask for
>> confirmation *if* it saw that this is a wildcard).
>>
>> 'Globbing' (if that means auto expansion) is taking away choice.
>
> And app globbing takes away choice, too. How would you, with the
> Windows 'del' command, remove a single file that has a question mark
> in the name? Or are you depending entirely on the fact that Windows
> doesn't let you do that? Here's some of my music collection:

I don't follow you. "?" is problematical on both systems. I think 
Windows disallows it completely: I get 'Protocol error' if I copy such a 
file from Linux to Windows. Presumably there is an escaping system, but 
I don't know what it is.

But we're talking about Linux. Obviously if a program is developed on 
Linux, can work on multiple files and DEPENDS on on being presented with 
that list of files as though it had been invoked as:

   program file1 file2 file3 file4 file5 file6 file7 .... fileN

then it will have problems if that feature is taken away. I can see how 
it can still be there after all these years!

-- 
Bartc


From marko at pacujo.net  Wed Dec  7 09:58:15 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 16:58:15 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87pol323co.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> I don't know any Unix programs that provide file spec processing.
> (That's not to say that there are absolutely none, only that I don't
> know of any.)

I can think of "find", "git" and "scp".

Git supports the recursive "**" wildcard, as well.


Marko


From p.f.moore at gmail.com  Wed Dec  7 10:02:04 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 7 Dec 2016 07:02:04 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <eaopp7Fj5v8U1@mid.individual.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me> <eaopp7Fj5v8U1@mid.individual.net>
Message-ID: <384e999d-6220-4b27-83ee-07c27de5afe8@googlegroups.com>

On Tuesday, 6 December 2016 21:44:18 UTC, Gregory Ewing  wrote:
> BartC wrote:
> > But those would be silly.
> > 
> > Some special syntax is known about: | < and > for example. % less so
> 
> What you need to understand is that, to a Unix user,
> * and ? are *just as well known* as |, < and >.

And to Windows users. Just because BartC is proposing a silly distinction between what "Unix users" and "Windows users" expect, doesn't mean there is such a distinction.

Users with limited experience of command line environments may well be unclear about special characters like * ? > < |. It's possibly even true that Windows users as a group are more likely to be unfamiliar with the command line than Unix users (although people using iPads and Android phones are technically "Unix users", so that point isn't as true as you'd expect).

But claiming that special syntax is an issue because people who are unfamiliar with it, are unfamiliar with it, is silly at best, and deliberate trolling at worst.

Paul


From rustompmody at gmail.com  Wed Dec  7 10:17:53 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Wed, 7 Dec 2016 07:17:53 -0800 (PST)
Subject: sets anomaly
Message-ID: <526563a5-4e1f-4098-a75f-d65282b4ef77@googlegroups.com>

Trying to write some code using sets (well frozen sets)
And was hit by this anomaly

This is the behavior of lists I analogously expect in sets:

>>> []
[]
>>> [[]]
[[]]
>>> 

ie the empty list and the list of the empty list are different things

However
(with
f= frozenset
)

>>> f()
frozenset()
>>> f([])
frozenset()
>>> f(f([]))
frozenset()
>>> 

Spent a good ? hour finding out this strangeness
And then some figuring out how to get an empty set into a set
This is the best I get:
>>> f([f([])])
frozenset({frozenset()})

Context:


Trying to write the simple combinatorics code which does the enumeration 
corresponding to the identity:
?Cr + ?Cr-1 = ???Cr

The Haskell code is very simple:
[`c` means function c used in infix]

[]      `c` (r+1) = []
xs      `c` 0	  = [[]]
(x::xs) `c` r	  = (xs `c` r) ++ [x::ys | ys <- xs `c` (r-1)]


And its usage:
? [1,2,3,4] `c` 2
[[3, 4], [2, 4], [2, 3], [1, 4], [1, 3], [1, 2]] : [[Int]]

I wanted to do it in python to discuss the question of sets and lists

The best I get is this. 

Can it be improved??


-------------------------------
f = frozenset

def c(zs,r):
    """ returns list of lists """
    if not zs and r > 0 : return []
    if r == 0 : return  [[]]
    x,xs = zs[0], zs[1:]
    return c(xs,r) + [[x]+ys for ys in c(xs,r-1)]

def css(zs,r):
    """ returns set of sets """
    if not zs and r > 0 : return f([])
    if r == 0 : return  f([f([])])
    x,xs = zs[0], zs[1:]
    return css(xs,r) | f([(f([x]) | ys) for ys in css(xs,r-1)])

def cls(zs,r):
    """ returns list of sets """
    if not zs and r > 0 : return []
    if r == 0 : return  [f([])]
    x,xs = zs[0], zs[1:]
    return cls(xs,r) + [f([x]) | ys for ys in cls(xs,r-1)]

And usage:

>>> c("ABC",2)
[['B', 'C'], ['A', 'C'], ['A', 'B']]
>>> css("ABC",2)
frozenset({frozenset({'C', 'A'}), frozenset({'C', 'B'}), frozenset({'B', 'A'})})
>>> cls("ABC",2)
[frozenset({'C', 'B'}), frozenset({'C', 'A'}), frozenset({'B', 'A'})]


From p.f.moore at gmail.com  Wed Dec  7 10:19:05 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 7 Dec 2016 07:19:05 -0800 (PST)
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o28t7e$2s9$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
Message-ID: <1533200a-8281-49e1-8e12-d40e3beec312@googlegroups.com>

On Wednesday, 7 December 2016 11:55:01 UTC, BartC  wrote:
> With automatic expansion, then EVERY program can become dangerous. Can 
> no one else see this?

Please start an elevated command prompt on your PC, and type the command

    rmdir /s /q C:\WINDOWS

There are no special characters here, so no possibility of "automatic expansion", so this is bound to be completely safe[1].

Can you not see that danger is unrelated to the existence of globbing? Danger is a result of people not knowing what the commands they type do.

Paul

[1] To anyone following along who isn't familiar with the command line, DON'T TRY THIS - IT WILL WRECK YOUR PC!


From bc at freeuk.com  Wed Dec  7 10:19:53 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 15:19:53 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2998b$ctm$1@dont-email.me>

On 07/12/2016 14:34, Steve D'Aprano wrote:
> On Wed, 7 Dec 2016 11:23 pm, BartC wrote:
>
>> On 07/12/2016 05:15, Steven D'Aprano wrote:
>>> On Wednesday 07 December 2016 12:55, BartC wrote:
>>>
>>>> But even Linux's 128KB will fill if someone wanted a command line that
>>>> listed 20,000 files individually. But it would be spectacularly bad use
>>>> of a command line interface which was designed for humans.
>>>
>>> That's EXACTLY the point of having the shell do globbing.
>>
>> Sorry, it's got NOTHING to do with it!
>>
>> Here's a command that works on all one million files in the current path:
>>
>>   program *.*

>> 11 characters, so Linux still has 131061 left (assuming 'program ' is
>> part of it).
>
> I don't know why you are so hung up over the number of characters here, or
> this bogeyman of "one million files" in a directory.

Because /you/ brought it up as a reason why 'globbing' would help when 
there are command limits?

  Who does such a thing?
> Most sys admins will go their entire career without ever needing to process
> a million files in a single directory, or even split over a dozen
> directories.

They probably never need to use a 128KB command line either.

> But... how does your program distinguish between the file spec *.* which
> should be expanded, and the literal file *.* which shouldn't be expanded?

Do you mean a *.* parameter that refers to all the files in the 
directory (whether the intention is to build a list of those files or 
not) and a *.* parameter that means something entirely different?

OK, but you go first: how does a Linux program tell the difference between:

   A B C

which signifies three files, and:

   A B C

which means two files, and an option C? And without using escapes!

> You need an escape mechanism too. You've done that, of course. Right?

My program wouldn't need anything so crude. The input syntax would be 
more carefully designed so as to not have the ambiguity.

And actually I generally avoid command line interfaces (ones used by 
shells) because they are such a pain. (Why, for example, aren't the 
contents of the command line - even after preprocessing by the shell - 
just read using exactly the same mechanism as subsequent lines of input?)

>>    list fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} \
>>      [a-z]/IMG* > files
>>
>>    command @files
>
> *Requiring* this would be completely unacceptable. Forcing the user to write
> two lines (one to create a variable, then to use it) every time they needed
> a glob expansion would go down among sys admins like a lead balloon.

Suppose there is a pattern that can't be expressed as a one-liner?

And the shell is supposed to be a full-fledged programming language, 
people keep saying. What can be more natural than to make use of 
variables? As you go on to demonstrate!

-- 
Bartc




From ben.usenet at bsb.me.uk  Wed Dec  7 10:25:57 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Wed, 07 Dec 2016 15:25:57 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
Message-ID: <874m2fg3qy.fsf@bsb.me.uk>

BartC <bc at freeuk.com> writes:
<snip>
> [...] But remember:
>
>   cp *.c
>
> There might be some irate users out there if it can't detect a simple
> user error like that.

There might be.  They are ill-served by current Unix shells.  But,
oddly, the flow has gone the other way.  People have wanted Unix shells
and utilities ported to Windows, not cmd.exe and friends ported to
Linux.  You clearly see a gap in the provision here.  Wouldn't it be
better for the world if you wrote cmd.exe for Linux rather than
complaining about what other shells do?

<snip>
> OK, I understand. Having a program launcher indiscriminately transform
> input A into A' is /much/ better than having it done by the program,
> even if the program doesn't want it and it is not meaningful.

Yes, you've explained that you understand the value of such
transformations, it's just that the ones you like happen to be exactly
those in cmd.exe.  You are very lucky that cmd.exe happened to get
exactly the right interpretation of the input or you would be raining
against /every/ shell on both systems.

<snip>
-- 
Ben.


From ned at nedbatchelder.com  Wed Dec  7 10:33:59 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Wed, 7 Dec 2016 07:33:59 -0800 (PST)
Subject: sets anomaly
In-Reply-To: <526563a5-4e1f-4098-a75f-d65282b4ef77@googlegroups.com>
References: <526563a5-4e1f-4098-a75f-d65282b4ef77@googlegroups.com>
Message-ID: <38edd97d-d515-4f69-8721-66e130f3e4b7@googlegroups.com>

On Wednesday, December 7, 2016 at 10:18:32 AM UTC-5, Rustom Mody wrote:
> Trying to write some code using sets (well frozen sets)
> And was hit by this anomaly
> 
> This is the behavior of lists I analogously expect in sets:
> 
> >>> []
> []
> >>> [[]]
> [[]]
> >>> 
> 
> ie the empty list and the list of the empty list are different things
> 
> However
> (with
> f= frozenset
> )
> 
> >>> f()
> frozenset()
> >>> f([])
> frozenset()
> >>> f(f([]))
> frozenset()
> >>> 

The difference is more about the difference between the behavior of a
callable constructor, and a list literal.  Lists will behave the same
as frozenset if you use list():

    >>> list()
    []
    >>> list(list())
    []
    >>> list(list(list()))
    []

This is because the constructors can take a sequence, and use those
elements as the new contents.  List literals don't work that way.


> Spent a good ? hour finding out this strangeness
> And then some figuring out how to get an empty set into a set
> This is the best I get:
> >>> f([f([])])
> frozenset({frozenset()})

That is the way I would have done it also. Or:

    >>> s = set()
    >>> s.add(frozenset())
    >>> frozenset(s)
    frozenset([frozenset([])])

Notice the repr output of the result shows how to make it! :)

--Ned.


From marko at pacujo.net  Wed Dec  7 10:43:33 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 07 Dec 2016 17:43:33 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <b403ba66-ac91-155d-1594-9d5af394ca82@gmail.com>
 <mailman.102.1481065258.2253.python-list@python.org>
 <877f7c2p3w.fsf@elektro.pacujo.net>
 <5847c11c$0$2912$c3e8da3$76491128@news.astraweb.com>
 <871sxk2gog.fsf@elektro.pacujo.net> <o295vk$qbe$1@blaine.gmane.org>
 <mailman.125.1481120588.2253.python-list@python.org>
Message-ID: <87k2bb2196.fsf@elektro.pacujo.net>

Grant Edwards <grant.b.edwards at gmail.com>:
> There are a few things in Unix that are fundamentally broken and
> really just can't be used for the things they are intended for (serial
> ports come to mind).
>
> However, that doesn't seem to prevent them from having been used
> sucessfully that way way for 40 years. ;)

Sigh. Those issues give me grief literally every day in the office.


Marko


From bc at freeuk.com  Wed Dec  7 10:48:32 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 15:48:32 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.124.1481119111.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
Message-ID: <o29au2$ip2$1@dont-email.me>

On 07/12/2016 13:58, Dennis Lee Bieber wrote:
> On Wed, 7 Dec 2016 11:54:35 +0000, BartC <bc at freeuk.com> declaimed the
> following:
>
>> With automatic expansion, then EVERY program can become dangerous. Can
>> no one else see this?
>>
>
> 	With your scheme, EVERY PROGRAM HAS TO IMPLEMENT ITS OWN GLOBBING.

Only the tiny minority that can be meaningfully invoked on an arbitrary 
number of files at the same time.

And globbing doesn't take care of all of it: a Linux program still has 
to iterate over a loop of filenames. The same as on Windows, except the 
latter will need to call a function to deliver the next filename.

> 	The result is BLOAT and INCONSISTANT BEHAVIOR.
>
> 	UNIX shell behavior is consist ant -- you learn the rules for ONE
> shell, and they apply to ALL programs started with that shell.
>
> 	You'll respond that a library should be provided for programs to do
> that... But a C library would have to be wrapped to be used in Python, Ada,

You make it sound like a big deal. Here's a program (in my language not 
Python) which prints the list of files matching a file-spec:

println dirlist(cmdparams[2])

It's invoked like this in Windows (pcc is interpreter, t is the program):

pcc t *.c

and like this in Linux:

./pcc t "*.c"

In both cases is displays a list of .c files. The Linux needs the quotes 
because otherwise it is expanded to a list of N files, of which the 
first is taken to be the filespec, so it will print only the first.

I would prefer that the program "t" can be invoked exactly the same way 
under both systems. I don't want different instructions for Linux, or 
for the user (of my language) to have to write two lots of code, as that 
is my job:

  println dirlist(cmdparams[2])        # Windows
  println tail(cmdparams)              # Linux

(Here are implementations of that dirlist() for both OSes: 
http://pastebin.com/knciRULE

Code is my dynamic language, not C. But you can see there is not much to 
it. No 3rd party libraries apart from what comes with the OS, or that is 
part of Posix.)

> what-have-you. Thereby one now has a possibly different API in each
> language for doing something that should be the same action. Or even
> multiple APIs in one language (just look at Python's library for command
> line argument parsing:

Just look at *any* of Python's thousands of libraries.

It's always seem strange that there are libraries to do any conceivable 
thing, but people here baulk at the basics (simple ways of reading stuff 
on a line, reading a single key, and now reading /the input to a 
program/, which is pretty important).

-- 
Bartc



From rosuav at gmail.com  Wed Dec  7 10:53:41 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 8 Dec 2016 02:53:41 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o297u6$7u0$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
 <mailman.119.1481114405.2253.python-list@python.org>
 <o297u6$7u0$1@dont-email.me>
Message-ID: <CAPTjJmrum6D7iVfjXpL+2y3q9HWLRXqizy90f7sn12apecC23w@mail.gmail.com>

On Thu, Dec 8, 2016 at 1:57 AM, BartC <bc at freeuk.com> wrote:
> On 07/12/2016 12:39, Chris Angelico wrote:
>>
>> On Wed, Dec 7, 2016 at 10:54 PM, BartC <bc at freeuk.com> wrote:
>>>
>>> But if a program existed that took N filename parameters with the purpose
>>> of
>>> deleting each of them, then it can't tell if they were typed in
>>> individually
>>> (so indicating a stronger intent), or whether a finger slipped on a
>>> single
>>> filename and a "*" was added, in which case it could choose to ask for
>>> confirmation *if* it saw that this is a wildcard).
>>>
>>> 'Globbing' (if that means auto expansion) is taking away choice.
>>
>>
>> And app globbing takes away choice, too. How would you, with the
>> Windows 'del' command, remove a single file that has a question mark
>> in the name? Or are you depending entirely on the fact that Windows
>> doesn't let you do that? Here's some of my music collection:
>
>
> I don't follow you. "?" is problematical on both systems. I think Windows
> disallows it completely: I get 'Protocol error' if I copy such a file from
> Linux to Windows. Presumably there is an escaping system, but I don't know
> what it is.

Nope. Not at all problematical on my Linux box. I can invoke that file
in any way I like, and it just works. Tab completion works. Globbing
works. Invoking it through Popen (or equivalent) works. It is a
perfectly acceptable file name.

The only time I have trouble with those files is when I copy them onto
an external drive so my brother can play them... on his Windows
computer. It's not the file name that has the problem. It's Windows.

ChrisA


From rosuav at gmail.com  Wed Dec  7 10:55:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 8 Dec 2016 02:55:52 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2998b$ctm$1@dont-email.me>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o2998b$ctm$1@dont-email.me>
Message-ID: <CAPTjJmqB72NPLcTBtQpq0bcNppreu6scNFj7XWEnLm3fKWOygg@mail.gmail.com>

On Thu, Dec 8, 2016 at 2:19 AM, BartC <bc at freeuk.com> wrote:
> OK, but you go first: how does a Linux program tell the difference between:
>
>   A B C
>
> which signifies three files, and:
>
>   A B C
>
> which means two files, and an option C? And without using escapes!

The option would be called "-C", which means you need only distinguish
between filenames *that begin with a hyphen* and options. Given that
this is fundamentally ambiguous, the solution has to be at the
application level, and by convention, you can say "-- A B -C" to mean
that these are all file names, or you can put an explicit path on it
"A B ./-C". Does a leading "./" count as an escape?

ChrisA


From skip.montanaro at gmail.com  Wed Dec  7 11:26:07 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 7 Dec 2016 10:26:07 -0600
Subject: "Best" websocket implementation for Python 2.x?
Message-ID: <CANc-5UxrY4ZwxW4rhTbXHbOHD5NscstgAZ3-Edk8TBCaDU-3zA@mail.gmail.com>

I've recently inherited a small bit of code which uses websockets. I think
the package it uses is Horoki Ohtani's package dating from 2010. I have v.
0.34.0. Looking on PyPI, I see another websocket package (not Ohtani's)
with a version of 0.2.1 (also dating from 2010, I believe).

These two packages would appear to have stalled. Looking around, I also
found

ws4py
websockets - Python 3.3 or later
autobahn.websocket

There are probably others I haven't yet stumbled upon. (I'd continue poking
around PyPI, but it seems the server is having problems.)

Any suggestions as to which version is "best?" I leave it to you to decide
what "best" means to you. Having never used websockets before, I'm not sure
what criteria to use.

Thx,

Skip


From torriem at gmail.com  Wed Dec  7 11:26:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 09:26:39 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <87pol323co.fsf@elektro.pacujo.net>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <87pol323co.fsf@elektro.pacujo.net>
Message-ID: <1622e348-bc20-314b-6089-ef70d05f6be8@gmail.com>

On 12/07/2016 07:58 AM, Marko Rauhamaa wrote:
> Steve D'Aprano <steve+python at pearwood.info>:
> 
>> I don't know any Unix programs that provide file spec processing.
>> (That's not to say that there are absolutely none, only that I don't
>> know of any.)
> 
> I can think of "find", "git" and "scp".
> 
> Git supports the recursive "**" wildcard, as well.

And rsync.


From karim.farokhnia at gmail.com  Wed Dec  7 11:38:51 2016
From: karim.farokhnia at gmail.com (Karim Farokhnia)
Date: Wed, 7 Dec 2016 08:38:51 -0800 (PST)
Subject: calling a program from Python batch file
Message-ID: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>

Hi there,

I am writing a batch file in Python. The batch file, in part, calls a program named "oq-console.bat" to run. Then once the program comes up (it looks like windows CMD), I need the batch file to type some commands to make it run (just like I would do it by myself). 

I need the batch file does the whole things automatically, I mean calls the program, types in commands and press enter!

So far, I have used the below Python codes in my batch file to do that;


import subprocess

subprocess.call(["C:\Program Files (x86)\OpenQuake Engine\oq-console.bat", "oq --run", "job.ini"])


Note that the first string is the program, and the last two are the commands which needs to be typed in.

It pulls up the program well, but doesn't type in the commands to make it to run.


Any ideas on how I can get the program to run the commands as well?

Thank you!


From skip.montanaro at gmail.com  Wed Dec  7 11:50:58 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 7 Dec 2016 10:50:58 -0600
Subject: "Best" websocket implementation for Python 2.x?
In-Reply-To: <CANc-5UxrY4ZwxW4rhTbXHbOHD5NscstgAZ3-Edk8TBCaDU-3zA@mail.gmail.com>
References: <CANc-5UxrY4ZwxW4rhTbXHbOHD5NscstgAZ3-Edk8TBCaDU-3zA@mail.gmail.com>
Message-ID: <CANc-5UyMOuXgqHqdY+PwL0CnZcZZsyTi4GsQy2FG-q2xW=gsPA@mail.gmail.com>

PyPI came back. A bit more sleuthing suggests that the
websocket-client package on PyPI is Ohtani's package, and is more
up-to-date than the copyright notices would suggest. The package was
updated a few days ago on GitHub.

Taking the path of least resistance (no changes necessary to the
code), I suspect I will just stick with that unless there are
overriding inputs from the community suggesting something else is way
better...

S


From torriem at gmail.com  Wed Dec  7 11:53:42 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 09:53:42 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29au2$ip2$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
Message-ID: <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>

On 12/07/2016 08:48 AM, BartC wrote:
> I would prefer that the program "t" can be invoked exactly the same way 
> under both systems. I don't want different instructions for Linux, or 
> for the user (of my language) to have to write two lots of code, as that 
> is my job...

Ahh now we come to the crux of your argument.  You want all potential
platforms to conform to your own idea of what is normal.  And indeed
this is the actual issue that started the original thread.  But that's
not the way it works for any platform.  I develop a lot of code on Linux
that I'd like to get running on Windows.  I prefer the Linux way of
doing things but I'm not going to make any headway if I just try to
brow-beat Windows and Windows' users over the things that aren't
implement the same way.  And unfortunately neither are you.  If I write
a script that's going to take some filenames on the command-line, I'm
going to have to implement a custom path to expand globs on Windows (if
that's the behavior I need).

You may prefer "t" to be invoked example the same way on Linux as
Windows, but unless your program feels natural on the Linux
command-line, it hardly matters to Linux users whether your program is
invoked in the exact same way on Linux and Windows.  If Linux is indeed
going to be a first-class target for an application, you will want to do
things the Linux or unix way.  This may mean disabling globbing
internally and just allowing arbitrary filenames on the command-line.
Whatever. But this strange idea of yours that every OS should be like
Windows is unrealistic and even highly offensive to some of our
sensibilities. It doesn't matter what is better or more logical in your
opinion.

Python may provide some nice abstractions, but at the end of the day,
there are always OS-specific things you'll have to and will want to deal
with.




From walters.justin01 at gmail.com  Wed Dec  7 12:15:21 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Wed, 7 Dec 2016 09:15:21 -0800
Subject: Django broken pipe error
In-Reply-To: <843bd5a8-f127-4fdc-8817-0b1645c1872d@googlegroups.com>
References: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
 <843bd5a8-f127-4fdc-8817-0b1645c1872d@googlegroups.com>
Message-ID: <CAO1D73H21KK14yheCLdH+LjXH_YWMaOoDM5Q7qDAm8kq1cS9QA@mail.gmail.com>

On Wed, Dec 7, 2016 at 1:08 AM, <dr.roman.graf at gmail.com> wrote:

> Thank you Justin,
>
> I'm on the dev server and should present results in this way.
>
> Yes, I use manage.py runserver --insecure to start the server (from
> PyCharm).
>
> My views.py call:
>
> @detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
> def export_report(request):
>     body_unicode = request.body.decode('utf-8')
>     body_str = body_unicode.encode('ascii','ignore')
>     attr_list = body_str.split('&')
>     attr_dict = {}
>     if (len(attr_list) > 0):
>         for attr in attr_list:
>            ...
>            key = key_value_pair[0]
>            attr_dict[key] = key_value_pair[1]
>     trend = trends.calculate_trend(
>         attr_dict['search_phrase']
>         , attr_dict['time_from']
>         , attr_dict['time_to']
>         , attr_dict['time_scale']
>     )
>     attr_dict['trend'] = trend
>     res = str(json.dumps(attr_dict))
>     return HttpResponse(res, content_type="text/plain; charset=utf-8")
>
> and trend calculation in trends.py with database calls:
>
> def calculate_trend(query_phrase, time_from, time_to, time_scale):
>     # check in database if trend already exists
>     try:
>         db_trend = Trend.objects.get(pk=query_phrase)
>         if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
>                 and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
>                 and db_trend.granularity == time_scale:
>             logger.info("trend already exists.")
>             existing_trend_dict = ast.literal_eval(db_trend.content)
>             return existing_trend_dict
>     except Trend.DoesNotExist:
>         logger.info("It is a new trend search.")
>     trend_dict = {}
>     start_time = pd.Timestamp(value[0])
>     end_time = pd.Timestamp(value[-1])
>     freq = ... get frequency using pandas lib
>     trend_dict[key] = freq
>     json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
>     trend = Trend(
>         phrase=query_phrase,
>         content=json_trend_content,
>         from_time=time_from,
>         to_time=time_to,
>         granularity=time_scale,
>     )
>     if trend is not None:
>         try:
>             db_trend = Trend.objects.get(pk=query_phrase)
>             db_trend.delete()
>             logger.info("delete old trend: %s. " % trend)
>         except Trend.DoesNotExist:
>             logger.info("create trend: %s. " % trend)
>         trend.save()
>     return trend_dict
>
> Thank you in advance!
>
> Roman
> --
> https://mail.python.org/mailman/listinfo/python-list
>


It looks like you can probably get rid of the try/except block at the end
of the calculate_trend
method as any existing Trend object will have already been caught in the
first try/except block.

>From what I'm reading here:
http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
,
this issue can be caused by the client closing the connection before
sendall() finishes writing. Can you estimate
how long it takes for a request to this endpoint takes to resolve? If it's
a long time(maybe due to the pandas call?),
your browser/client may be timing out. It could be because it takes a while
for the Db to find an existing Trend object
as well.

I can't give you any advice on how to fix it exactly, but I can tell you
what the problem is: The client is closing the
connection before socket.sendall() has finished writing to the socket. My
guess is that the calculate_trend() method
takes a long time to complete and the client is timing out.


From torriem at gmail.com  Wed Dec  7 12:18:33 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 10:18:33 -0700
Subject: calling a program from Python batch file
In-Reply-To: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
Message-ID: <d5822889-1cbf-bc20-c41b-76d8048ad429@gmail.com>

On 12/07/2016 09:38 AM, Karim Farokhnia wrote:
> I am writing a batch file in Python. The batch file, in part, calls a
> program named "oq-console.bat" to run. Then once the program comes up
> (it looks like windows CMD), I need the batch file to type some
> commands to make it run (just like I would do it by myself).
> 
> I need the batch file does the whole things automatically, I mean
> calls the program, types in commands and press enter!

Is the program that oq-console.bat runs interactive?  After it launches
into its own console window it asks for more input?

> So far, I have used the below Python codes in my batch file to do
> that;
> 
> 
> import subprocess
> 
> subprocess.call(["C:\Program Files (x86)\OpenQuake
> Engine\oq-console.bat", "oq --run", "job.ini"])
> 
> 
> Note that the first string is the program, and the last two are the
> commands which needs to be typed in.
> 
> It pulls up the program well, but doesn't type in the commands to
> make it to run.

You'll have to explain what you mean by "[it] doesn't type ni the
commands to make it run."



From torriem at gmail.com  Wed Dec  7 12:19:27 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 10:19:27 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o297u6$7u0$1@dont-email.me>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
 <mailman.119.1481114405.2253.python-list@python.org>
 <o297u6$7u0$1@dont-email.me>
Message-ID: <aae1c84e-5311-773b-da7d-eab2f43328c5@gmail.com>

On 12/07/2016 07:57 AM, BartC wrote:
> I don't follow you. "?" is problematical on both systems. I think 
> Windows disallows it completely: I get 'Protocol error' if I copy such a 
> file from Linux to Windows. Presumably there is an escaping system, but 
> I don't know what it is.

As Chris says, no it's not problematical. Only / and null are not
allowed in filenames on Unix (Posix) systems including Linux.  Many
filename characters do have special meaning to shells, so they must be
escaped when specifying them on a shell prompt (but they don't have to
be escaped if another program is calling the program with popen).

> But we're talking about Linux. Obviously if a program is developed on 
> Linux, can work on multiple files and DEPENDS on on being presented with 
> that list of files as though it had been invoked as:
> 
>    program file1 file2 file3 file4 file5 file6 file7 .... fileN
> 
> then it will have problems if that feature is taken away. I can see how 
> it can still be there after all these years!

40 years is a long time, plenty of time to change what doesn't work.

This idea of specifying multiple files individually on a command-line
not only works, it's there for good reason.  I've alluded to it several
times. This idea is part of how we can string together small scripts to
do batch processing and filtering (of data or files or whatever) using
the shell.  You could say it's part of the interprocess communication
protocol that shell scripting relies upon.  Globbing by each individual
app would break this protocol and make it harder to string these bits
and bobs together.  This protocol involves piping of standard in and
standard out, or to pipe the output from one command onto the
command-line of another app (multiple channels).  This is the power of
the unix shell and this is why I and many others use it for our daily
work every day.  And this is why these concepts that trip you up have
had such staying power.  We simply haven't found a better way of doing
it.  Also the unix shell protocol and ideal allows you to completely
replace the shell with something else, and the scripts processes one has
developed still function!  C Shell, Bourne shell, Bash, zsh, fish.  Just
pick the one you want (each with their own quirks and expansion rules
and quoting and escaping rules).

I've seen attempts as using something based on Python to form a shell,
but ultimately they never feel quite right to me.  Mainly because
there's an impedance mismatch between python's objects and files on a
filesystem. Does such an impedance mismatch come with PowerShell too? I
dunno.



From walters.justin01 at gmail.com  Wed Dec  7 12:23:05 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Wed, 7 Dec 2016 09:23:05 -0800
Subject: "Best" websocket implementation for Python 2.x?
In-Reply-To: <CANc-5UyMOuXgqHqdY+PwL0CnZcZZsyTi4GsQy2FG-q2xW=gsPA@mail.gmail.com>
References: <CANc-5UxrY4ZwxW4rhTbXHbOHD5NscstgAZ3-Edk8TBCaDU-3zA@mail.gmail.com>
 <CANc-5UyMOuXgqHqdY+PwL0CnZcZZsyTi4GsQy2FG-q2xW=gsPA@mail.gmail.com>
Message-ID: <CAO1D73EgRNPhHCVqfv+CPJcpA+dJgM04cz8RGikLCNyXKfVmng@mail.gmail.com>

On Wed, Dec 7, 2016 at 8:50 AM, Skip Montanaro <skip.montanaro at gmail.com>
wrote:

> PyPI came back. A bit more sleuthing suggests that the
> websocket-client package on PyPI is Ohtani's package, and is more
> up-to-date than the copyright notices would suggest. The package was
> updated a few days ago on GitHub.
>
> Taking the path of least resistance (no changes necessary to the
> code), I suspect I will just stick with that unless there are
> overriding inputs from the community suggesting something else is way
> better...
>
> S
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I can't vouch for it being better, but if you're using asgi, you may want
to take a look at Daphne: https://github.com/django/daphne.

I believe it uses autobahn to power the websocket side of things. Daphne
was developed for the django-channels project. I have
used channels before and I can say that it works pretty well.


From bgailer at gmail.com  Wed Dec  7 12:31:28 2016
From: bgailer at gmail.com (Bob Gailer)
Date: Wed, 7 Dec 2016 12:31:28 -0500
Subject: calling a program from Python batch file
In-Reply-To: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
Message-ID: <CAP1rxO5XMSWOoN_ZXjhD-kBXM+0V2VTM6op4HEU1q8iig57zFw@mail.gmail.com>

On Dec 7, 2016 11:40 AM, "Karim Farokhnia" <karim.farokhnia at gmail.com>
wrote:
>
> Hi there,
>
> I am writing a batch file in Python. The batch file, in part, calls a
program named "oq-console.bat" to run. Then once the program comes up (it
looks like windows CMD), I need the batch file to type some commands to
make it run (just like I would do it by myself).
>
> I need the batch file does the whole things automatically, I mean calls
the program, types in commands and press enter!

Try using the stdin argument of subprocess.
>
> So far, I have used the below Python codes in my batch file to do that;
>
>
> import subprocess
>
> subprocess.call(["C:\Program Files (x86)\OpenQuake
Engine\oq-console.bat", "oq --run", "job.ini"])
>
>
> Note that the first string is the program, and the last two are the
commands which needs to be typed in.
>
> It pulls up the program well, but doesn't type in the commands to make it
to run.
>
>
> Any ideas on how I can get the program to run the commands as well?
>
> Thank you!
> --
> https://mail.python.org/mailman/listinfo/python-list


From karim.farokhnia at gmail.com  Wed Dec  7 12:43:04 2016
From: karim.farokhnia at gmail.com (Karim Farokhnia)
Date: Wed, 7 Dec 2016 09:43:04 -0800 (PST)
Subject: calling a program from Python batch file
In-Reply-To: <mailman.133.1481131120.2253.python-list@python.org>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
 <d5822889-1cbf-bc20-c41b-76d8048ad429@gmail.com>
 <mailman.133.1481131120.2253.python-list@python.org>
Message-ID: <0454718e-2684-49cc-af13-7195a390ba0d@googlegroups.com>

On Wednesday, December 7, 2016 at 12:18:51 PM UTC-5, Michael Torrie wrote:
> On 12/07/2016 09:38 AM, Karim Farokhnia wrote:
> > I am writing a batch file in Python. The batch file, in part, calls a
> > program named "oq-console.bat" to run. Then once the program comes up
> > (it looks like windows CMD), I need the batch file to type some
> > commands to make it run (just like I would do it by myself).
> > 
> > I need the batch file does the whole things automatically, I mean
> > calls the program, types in commands and press enter!
> 
> Is the program that oq-console.bat runs interactive?  After it launches
> into its own console window it asks for more input?
The program doesn't ask for more input parameters as all parameters are in input files and the program will automatically read the files, but I need to type in some required commands and press enter.

> 
> > So far, I have used the below Python codes in my batch file to do
> > that;
> > 
> > 
> > import subprocess
> > 
> > subprocess.call(["C:\Program Files (x86)\OpenQuake
> > Engine\oq-console.bat", "oq --run", "job.ini"])
> > 
> > 
> > Note that the first string is the program, and the last two are the
> > commands which needs to be typed in.
> > 
> > It pulls up the program well, but doesn't type in the commands to
> > make it to run.
> 
> You'll have to explain what you mean by "[it] doesn't type ni the
> commands to make it run."

By [it] I meant the program.


From torriem at gmail.com  Wed Dec  7 12:58:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 10:58:39 -0700
Subject: calling a program from Python batch file
In-Reply-To: <0454718e-2684-49cc-af13-7195a390ba0d@googlegroups.com>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
 <d5822889-1cbf-bc20-c41b-76d8048ad429@gmail.com>
 <mailman.133.1481131120.2253.python-list@python.org>
 <0454718e-2684-49cc-af13-7195a390ba0d@googlegroups.com>
Message-ID: <703c0d43-b620-1051-ae3b-61ebacaff50a@gmail.com>

On 12/07/2016 10:43 AM, Karim Farokhnia wrote:
>> Is the program that oq-console.bat runs interactive?  After it
>> launches into its own console window it asks for more input?
> The program doesn't ask for more input parameters as all parameters
> are in input files and the program will automatically read the files,
> but I need to type in some required commands and press enter.

think what you mean is that it *is* interactive, at least requiring some
input before it begins to work.

As the other person who replied said, it sounds like this program will
accept input on standard in.  From python you will probably have to use
subprocess.popen and use the .communicate() method.  You will be able to
pass in a string that will get fed to the subprocess on it's standard in
pipe (the equivalent of typing and pressing enter). Check the online
docs, particularly for the .communicate() method.



From gordon at panix.com  Wed Dec  7 12:59:39 2016
From: gordon at panix.com (John Gordon)
Date: Wed, 7 Dec 2016 17:59:39 +0000 (UTC)
Subject: calling a program from Python batch file
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
Message-ID: <o29imb$h3i$1@reader1.panix.com>

In <f9d6e83d-d8e7-4245-a02f-9065572632a9 at googlegroups.com> Karim Farokhnia <karim.farokhnia at gmail.com> writes:

> Hi there,

> I am writing a batch file in Python. The batch file, in part, calls a
> program named "oq-console.bat" to run. Then once the program comes up
> (it looks like windows CMD), I need the batch file to type some commands
> to make it run (just like I would do it by myself).

If you need to start up a program, provide interactive input to it, and
perhaps examine its interactive output, then you want the "pexpect" module:

    Pexpect is a pure Python module for spawning child applications;
    controlling them; and responding to expected patterns in their output.
    Pexpect allows your script to spawn a child application and control it
    as if a human were typing commands.

    https://pexpect.readthedocs.io/en/stable/

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From bc at freeuk.com  Wed Dec  7 13:02:21 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 18:02:21 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.131.1481129628.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
Message-ID: <o29ip0$g0v$1@dont-email.me>

On 07/12/2016 16:53, Michael Torrie wrote:
> On 12/07/2016 08:48 AM, BartC wrote:
>> I would prefer that the program "t" can be invoked exactly the same way
>> under both systems. I don't want different instructions for Linux, or
>> for the user (of my language) to have to write two lots of code, as that
>> is my job...
>
> Ahh now we come to the crux of your argument.  You want all potential
> platforms to conform to your own idea of what is normal.  And indeed
> this is the actual issue that started the original thread.  But that's
> not the way it works for any platform.  I develop a lot of code on Linux
> that I'd like to get running on Windows.  I prefer the Linux way of
> doing things but I'm not going to make any headway if I just try to
> brow-beat Windows and Windows' users over the things that aren't
> implement the same way.

There is a lack of balance.

If I run a Linux program on Windows, then the application gets to see 
parameters such as *.* and can expand them if it wants.

If I want to run a Windows program on Linux, and that program needs to 
see *.* unexpanded, then it can't undo that expansion. The cat is 
already out of the bag.

It seems the only choice, if someone wants a cross-platform application 
that is invoked in the same way, is to dumb it down and make it assume 
that any parameters such as A* have been expanded, or will be expanded 
by that extra step.

Which means that input of A B* *C will all end up running together so 
that you have no idea what is what or which file corresponds to which 
expansion. That's assuming these parameters have any connection to the 
current set of files at all; if not, then the input you end up is going 
to be meaningless.

Now the problem is to determine whether processed input of U V W X Y Z 
are intended bona fide inputs, or whether they just happened to result 
from some random associations with the files in the current directory.

But this is what happens when you have to work to the lowest common 
denominator. The only proper solution is ban any unescaped ? or * from 
inputs, and to add a warning that the program could behave unexpectedly 
if they are used inadvertently.

(I'm in the middle of porting my console editor to Linux. But one 
problem is that on one Linux, half the key combinations (eg. 
Shift+Ctrl+B) are not recognised. On other Linux, nearly all of those 
are missing too, plus most of the rest! Which means I have to adapt the 
editor, again, to the lowest common denominator, the minimum set of keys 
that are guaranteed to work on both.

Except that was only two Linuxes; perhaps on others, the keyboard will 
likely be crippled in some other way. That is, the same universal USB 
keyboard that you can buy anywhere for $5 or $10, which has keys marked 
Shift, Ctrl and Alt, but whose combinations cannot be recognised 
consistently.

How people manage to do anything on such an OS I've no idea. Or maybe 
they spend all day just typing elaborate file-matching patterns to pipe 
between applications.)

-- 
Bartc


From amamaenko at gmail.com  Wed Dec  7 13:17:08 2016
From: amamaenko at gmail.com (Anton Mamaenko)
Date: Wed, 7 Dec 2016 21:17:08 +0300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29ip0$g0v$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me>
Message-ID: <28877D50-0E7D-473D-AC44-6FBE99712F3E@gmail.com>

Wow... this thread gets to finally become a holy war.  One method is dealing with this is take the environment as granted (Windows/Unix/Linux/MacOS/Dos sucks while my 
Windows/Unix/Linux/MacOS/Dos is the best) and figure if the hassle of porting (time, tech, AND moral) worth it. He just "do not try. Do or not do."

So stop complaining. Just live with it. 

Regards,
       Anton

> On 7 Dec 2016, at 21:02, BartC <bc at freeuk.com> wrote:
> 
>> On 07/12/2016 16:53, Michael Torrie wrote:
>>> On 12/07/2016 08:48 AM, BartC wrote:
>>> I would prefer that the program "t" can be invoked exactly the same way
>>> under both systems. I don't want different instructions for Linux, or
>>> for the user (of my language) to have to write two lots of code, as that
>>> is my job...
>> 
>> Ahh now we come to the crux of your argument.  You want all potential
>> platforms to conform to your own idea of what is normal.  And indeed
>> this is the actual issue that started the original thread.  But that's
>> not the way it works for any platform.  I develop a lot of code on Linux
>> that I'd like to get running on Windows.  I prefer the Linux way of
>> doing things but I'm not going to make any headway if I just try to
>> brow-beat Windows and Windows' users over the things that aren't
>> implement the same way.
> 
> There is a lack of balance.
> 
> If I run a Linux program on Windows, then the application gets to see parameters such as *.* and can expand them if it wants.
> 
> If I want to run a Windows program on Linux, and that program needs to see *.* unexpanded, then it can't undo that expansion. The cat is already out of the bag.
> 
> It seems the only choice, if someone wants a cross-platform application that is invoked in the same way, is to dumb it down and make it assume that any parameters such as A* have been expanded, or will be expanded by that extra step.
> 
> Which means that input of A B* *C will all end up running together so that you have no idea what is what or which file corresponds to which expansion. That's assuming these parameters have any connection to the current set of files at all; if not, then the input you end up is going to be meaningless.
> 
> Now the problem is to determine whether processed input of U V W X Y Z are intended bona fide inputs, or whether they just happened to result from some random associations with the files in the current directory.
> 
> But this is what happens when you have to work to the lowest common denominator. The only proper solution is ban any unescaped ? or * from inputs, and to add a warning that the program could behave unexpectedly if they are used inadvertently.
> 
> (I'm in the middle of porting my console editor to Linux. But one problem is that on one Linux, half the key combinations (eg. Shift+Ctrl+B) are not recognised. On other Linux, nearly all of those are missing too, plus most of the rest! Which means I have to adapt the editor, again, to the lowest common denominator, the minimum set of keys that are guaranteed to work on both.
> 
> Except that was only two Linuxes; perhaps on others, the keyboard will likely be crippled in some other way. That is, the same universal USB keyboard that you can buy anywhere for $5 or $10, which has keys marked Shift, Ctrl and Alt, but whose combinations cannot be recognised consistently.
> 
> How people manage to do anything on such an OS I've no idea. Or maybe they spend all day just typing elaborate file-matching patterns to pipe between applications.)
> 
> -- 
> Bartc
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From torriem at gmail.com  Wed Dec  7 13:23:02 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 11:23:02 -0700
Subject: calling a program from Python batch file
In-Reply-To: <o29imb$h3i$1@reader1.panix.com>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
 <o29imb$h3i$1@reader1.panix.com>
Message-ID: <033590e2-2717-76e2-73a0-462df35280ec@gmail.com>

On 12/07/2016 10:59 AM, John Gordon wrote:
> In <f9d6e83d-d8e7-4245-a02f-9065572632a9 at googlegroups.com> Karim Farokhnia <karim.farokhnia at gmail.com> writes:
> 
>> Hi there,
> 
>> I am writing a batch file in Python. The batch file, in part, calls a
>> program named "oq-console.bat" to run. Then once the program comes up
>> (it looks like windows CMD), I need the batch file to type some commands
>> to make it run (just like I would do it by myself).
> 
> If you need to start up a program, provide interactive input to it, and
> perhaps examine its interactive output, then you want the "pexpect" module:
> 
>     Pexpect is a pure Python module for spawning child applications;
>     controlling them; and responding to expected patterns in their output.
>     Pexpect allows your script to spawn a child application and control it
>     as if a human were typing commands.
> 
>     https://pexpect.readthedocs.io/en/stable/

Does Pexpect work on Windows?

In the OP's case it looks like the standard in pipe is sufficient.



From python at mrabarnett.plus.com  Wed Dec  7 13:26:17 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 7 Dec 2016 18:26:17 +0000
Subject: sets anomaly
In-Reply-To: <38edd97d-d515-4f69-8721-66e130f3e4b7@googlegroups.com>
References: <526563a5-4e1f-4098-a75f-d65282b4ef77@googlegroups.com>
 <38edd97d-d515-4f69-8721-66e130f3e4b7@googlegroups.com>
Message-ID: <bb85ddfb-365d-7b63-460f-d3fd01c9f6af@mrabarnett.plus.com>

On 2016-12-07 15:33, Ned Batchelder wrote:
> On Wednesday, December 7, 2016 at 10:18:32 AM UTC-5, Rustom Mody wrote:
>> Trying to write some code using sets (well frozen sets)
>> And was hit by this anomaly
>>
>> This is the behavior of lists I analogously expect in sets:
>>
>> >>> []
>> []
>> >>> [[]]
>> [[]]
>> >>>
>>
>> ie the empty list and the list of the empty list are different things
>>
>> However
>> (with
>> f= frozenset
>> )
>>
>> >>> f()
>> frozenset()
>> >>> f([])
>> frozenset()
>> >>> f(f([]))
>> frozenset()
>> >>>
>
> The difference is more about the difference between the behavior of a
> callable constructor, and a list literal.  Lists will behave the same
> as frozenset if you use list():
>
>     >>> list()
>     []
>     >>> list(list())
>     []
>     >>> list(list(list()))
>     []
>
> This is because the constructors can take a sequence, and use those
> elements as the new contents.  List literals don't work that way.
>
>
>> Spent a good ? hour finding out this strangeness
>> And then some figuring out how to get an empty set into a set
>> This is the best I get:
>> >>> f([f([])])
>> frozenset({frozenset()})
>
> That is the way I would have done it also. Or:
>
>     >>> s = set()
>     >>> s.add(frozenset())
>     >>> frozenset(s)
>     frozenset([frozenset([])])
>
> Notice the repr output of the result shows how to make it! :)
>
You must be using Python 2 because on Python 3 I get:

 >>> frozenset([frozenset()])
frozenset({frozenset()})



From bc at freeuk.com  Wed Dec  7 13:26:42 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 18:26:42 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <874m2fg3qy.fsf@bsb.me.uk>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me> <874m2fg3qy.fsf@bsb.me.uk>
Message-ID: <o29k6l$l4f$1@dont-email.me>

On 07/12/2016 15:25, Ben Bacarisse wrote:
> BartC <bc at freeuk.com> writes:
> <snip>
>> [...] But remember:
>>
>>   cp *.c
>>
>> There might be some irate users out there if it can't detect a simple
>> user error like that.
>
> There might be.  They are ill-served by current Unix shells.  But,
> oddly, the flow has gone the other way.  People have wanted Unix shells
> and utilities ported to Windows, not cmd.exe and friends ported to
> Linux.  You clearly see a gap in the provision here.  Wouldn't it be
> better for the world if you wrote cmd.exe for Linux rather than
> complaining about what other shells do?

It's not my job to write OSes. When I started doing this stuff, either 
there was no OS, or there was just a minimal one providing file services 
and allowing you to launch applications.

Now it seems to have trouble doing even that!

(As for people wanting Linux, I don't know why that is. Maybe Linux is 
used more in colleges and people want to continue working the same way, 
maybe Linux is perceived as free and Windows a product of evil 
capitalist M$. Your guess is as good as mine.

But you might have seen my objections; even putting aside bias in favour 
of one OS or another, I think many of them are valid. That fact Linux is 
getting more popular does not mean it is always better.)

>> OK, I understand. Having a program launcher indiscriminately transform
>> input A into A' is /much/ better than having it done by the program,
>> even if the program doesn't want it and it is not meaningful.
>
> Yes, you've explained that you understand the value of such
> transformations, it's just that the ones you like happen to be exactly
> those in cmd.exe.  You are very lucky that cmd.exe happened to get
> exactly the right interpretation of the input or you would be raining
> against /every/ shell on both systems.

I can do that as well if you like!

But it just seems odd in that I've complained in the past about Python's 
method for reading interactive lines of input.

Apparently, having line input presented in one lump (a single string) 
which is then explicitly chopped up by the program into tokens, is far 
superior to any other scheme, including any of mine.

But here, having input from the command line presented as a 
ready-separated set of arguments, and with (possibly meaningless) 
expansion into multiple files to boot, is far superior to getting the 
input in one lump (a single string).

So for one kind of line input, Black is better than White; for the 
other, White is better than Black!

-- 
Bartc


From stephen_tucker at sil.org  Wed Dec  7 14:55:34 2016
From: stephen_tucker at sil.org (Stephen Tucker)
Date: Wed, 7 Dec 2016 19:55:34 +0000
Subject: calling a program from Python batch file
In-Reply-To: <033590e2-2717-76e2-73a0-462df35280ec@gmail.com>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
 <o29imb$h3i$1@reader1.panix.com>
 <033590e2-2717-76e2-73a0-462df35280ec@gmail.com>
Message-ID: <CAP=-cKWCVy7zjY_nwB2_vULV1Y9Q4ift_bRj8=CRGZEdS5Vo1g@mail.gmail.com>

This might be totally irrelevant, but, if (a) the data to be read by the
program during a given run is known when the program is run/launched, (b)
that data is purely textual and (c) that data can be read by the program
from the stdin channel, then my idea is (1) before the launch, put that
data into a file named, say, inputfile.dat, and then (2)  launch the
program with a command like

call program.bat <inputfile.dat

Moreover, if the program needs to output textual data, and can do so using
stdout, then the data can be sent to another file named, say,
outputfile.dat as follows:

call program.bat <inputfile.dat >outputfile.dat

Finally, if the program can read its data from a given file and possibly
needs to output its data, again, to a named file, and those files' names
can be read from the parameters to the call, then

call program.bat inputfile.dat outputfile.dat

can be used.








On Wed, Dec 7, 2016 at 6:23 PM, Michael Torrie <torriem at gmail.com> wrote:

> On 12/07/2016 10:59 AM, John Gordon wrote:
> > In <f9d6e83d-d8e7-4245-a02f-9065572632a9 at googlegroups.com> Karim
> Farokhnia <karim.farokhnia at gmail.com> writes:
> >
> >> Hi there,
> >
> >> I am writing a batch file in Python. The batch file, in part, calls a
> >> program named "oq-console.bat" to run. Then once the program comes up
> >> (it looks like windows CMD), I need the batch file to type some commands
> >> to make it run (just like I would do it by myself).
> >
> > If you need to start up a program, provide interactive input to it, and
> > perhaps examine its interactive output, then you want the "pexpect"
> module:
> >
> >     Pexpect is a pure Python module for spawning child applications;
> >     controlling them; and responding to expected patterns in their
> output.
> >     Pexpect allows your script to spawn a child application and control
> it
> >     as if a human were typing commands.
> >
> >     https://pexpect.readthedocs.io/en/stable/
>
> Does Pexpect work on Windows?
>
> In the OP's case it looks like the standard in pipe is sufficient.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From bc at freeuk.com  Wed Dec  7 14:57:50 2016
From: bc at freeuk.com (BartC)
Date: Wed, 7 Dec 2016 19:57:50 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29ip0$g0v$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me>
Message-ID: <6RZ1A.155334$EX4.96179@fx22.am4>

On 07/12/2016 18:02, BartC wrote:

> If I want to run a Windows program on Linux, and that program needs to
> see *.* unexpanded, then it can't undo that expansion. The cat is
> already out of the bag.

I mean, the genie is out of the bottle, whatever. You know what I mean, 
the operation is not reversible.

-- 
bartc


From skip.montanaro at gmail.com  Wed Dec  7 15:01:10 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 7 Dec 2016 14:01:10 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <28877D50-0E7D-473D-AC44-6FBE99712F3E@gmail.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <28877D50-0E7D-473D-AC44-6FBE99712F3E@gmail.com>
Message-ID: <CANc-5Ux6MLfQ++pMGtYC5MqSCQqr=N-cvTyO0sFh8vhYV4DMLA@mail.gmail.com>

On Wed, Dec 7, 2016 at 12:17 PM, Anton Mamaenko <amamaenko at gmail.com> wrote:
> Wow... this thread gets to finally become a holy war.

It's not like this particular holy war is new, either. I'm sure it was
beaten to death in the days before Windows was a thing. As I indicated
in an earlier note, VMS and Unix did command line parsing (well,
globbing at least) differently. There were holy wars about that on
Usenet before many here were born, before many more were out of grade
school. Add to that the holy wars between Apollo and Sun Microsystems
(network topology (Token Ring v. Ethernet), as I recall, probably
remote procedure calls and networked file systems as well). I remember
being at one meeting where the Apollo and Sun gangs were going at it
hammer-and-tong, in person, not with the anonymity of the Internet to
obscure their views of each other. I can't now remember if that was
about RPC or networked file systems. Those were the good old days. :-)

Some things are better just left alone. File globbing is probably one
of those things. There's no right answer, and the people in the two
camps will never come to a compromise.

Skip


From eryksun at gmail.com  Wed Dec  7 15:17:02 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 7 Dec 2016 20:17:02 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
Message-ID: <CACL+1avqbbF1CeVat88i0ghFqW=W=xJrEbHdEGJmEFuS4KDaNg@mail.gmail.com>

On Wed, Dec 7, 2016 at 12:39 PM, Chris Angelico <rosuav at gmail.com> wrote:
> Note that two of the Beauty Stone tracks include quotes as well as
> question marks. How do you identify those? Let's say you want to play
> one of these in VLC, and then maybe you decide that the track in
> Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild
> it from the one in the parent directory. How does that work on
> Windows? If you say "it doesn't", then (a) you have taken away choice
> on a fundamental level, (b) you have your head in the sand, and (c)
> you still haven't solved the problem of percent signs, carets, and so
> on, which are perfectly legal in file names, but have meaning to the
> shell.

The five wildcard characters ("<>*?) aren't allowed in the names of
files and directories -- at least not by any Windows filesystem that
I've used. This makes it easy for a filesystem to support globbing in
its implementation of NtQueryDirectoryFile. Filenames also can't
contain control characters, slash, backslash, pipe, and colon (the
latter delimits a fully-qualified NTFS name, e.g.
filename:streamname:streamtype). NTFS stream names are less limited.
They only disallow NUL, slash, backslash, and colon.

The filesystem runtime library provides the macro
FsRtlIsAnsiCharacterLegal [1], among other related macros, which
allows filesystem drivers to be consistent with FAT or NTFS. To my
knowledge this is voluntary, but going against the grain is only
asking for headaches.

[1]: https://msdn.microsoft.com/en-us/library/ff546731

This macro depends on the array FsRtlLegalAnsiCharacterArray, which
indicates whether each ASCII character is valid for a fixed set of
filesystems. The flag values are as follows:

    0x01 - FAT
    0x02 - OS/2 HPFS
    0x04 - NTFS/Stream
    0x08 - Wildcard
    0x10 - Stream

Here's the array dumped from the kernel debugger. For convenience I've
added the printable ASCII characters above each line.

    lkd> db poi(nt!FsRtlLegalAnsiCharacterArray)

    fffff801`fc0e8550  00 10 10 10 10 10 10 10-10 10 10 10 10 10 10 10

    fffff801`fc0e8560  10 10 10 10 10 10 10 10-10 10 10 10 10 10 10 10
                           !  "  #  $  %  &  '  (  )  *  +  ,  -  .  /
    fffff801`fc0e8570  17 07 18 17 17 17 17 17-17 17 18 16 16 17 07 00
                        0  1  2  3  4  5  6  7  8  9  :  ;  <  =  >  ?
    fffff801`fc0e8580  17 17 17 17 17 17 17 17-17 17 04 16 18 16 18 18
                        @  A  B  C  D  E  F  G  H  I  J  K  L  M  N  O
    fffff801`fc0e8590  17 17 17 17 17 17 17 17-17 17 17 17 17 17 17 17
                        P  Q  R  S  T  U  V  W  X  Y  Z  [  \  ]  ^  _
    fffff801`fc0e85a0  17 17 17 17 17 17 17 17-17 17 17 16 00 16 17 17
                        `  a  b  c  d  e  f  g  h  i  j  k  l  m  n  o
    fffff801`fc0e85b0  17 17 17 17 17 17 17 17-17 17 17 17 17 17 17 17
                        p  q  r  s  t  u  v  w  x  y  z  {  |  }  ~
    fffff801`fc0e85c0  17 17 17 17 17 17 17 17-17 17 17 17 10 17 17 17

NTFS stream names are the least restricted, allowing everything except
NUL, slash, and backslash. For example:

    >>> open('test:\x01|*?<>"', 'w').close()
    >>> win32file.FindStreams('test')
    [(0, '::$DATA'), (0, ':\x01|*?<>":$DATA')]

The first stream listed above is the anonymous data stream
"test::$DATA", which is the same as simply opening "test".

Technically by the above table ":" is allowed in NTFS names, but its
use is reserved as the delimiter of the fully-qualified name. For
example, the fully-qualified name of a directory is
"dirname:$I30:$INDEX_ALLOCATION". The stream name is "$I30" and the
stream type is "$INDEX_ALLOCATION". A directory can also have multiple
named $DATA streams, because NTFS is weird like that.


From lew.pitcher at digitalfreehold.ca  Wed Dec  7 15:29:49 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Wed, 07 Dec 2016 15:29:49 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <6RZ1A.155334$EX4.96179@fx22.am4>
Message-ID: <o29rde$ft9$1@dont-email.me>

On Wednesday December 7 2016 14:57, in comp.lang.python, "BartC"
<bc at freeuk.com> wrote:

> On 07/12/2016 18:02, BartC wrote:
> 
>> If I want to run a Windows program on Linux, and that program needs to
>> see *.* unexpanded, then it can't undo that expansion. The cat is
>> already out of the bag.
> 
> I mean, the genie is out of the bottle, whatever. You know what I mean,
> the operation is not reversible.

So, don't pull the cork.

Look, Bart, in a Unix shell, the user has the option to enable or disable
globbing (that action that converts the argument * into a list of files). A
Unix shell can give a program exactly the same list of unaltered arguments
that a Windows shell can.

But, point of fact is that the feature to disable globbing is not often
needed. Most Unix programs that accept filenames are happy to accept a list
of filenames. There is not much call for a program to perform it's own
globbing, like is required in Windows.

In fact, I can only think of three Unix "commandline" programs that need shell
globbing disabled:
  find - which performs it's own filename matching
  grep - which uses regular expressions to search the contents of files, and
  sed  - which uses regular expressions to edit the contents of files.
(I'm sure that there are a few more examples, though).

If your argument is with Unix shell globbing, I say that you can disable it.

If your argument is with Python's use of the default shell options to parse
commandlines in the Popen() method, then you need to talk to the authors of
Python to see if they can give you a way to alter this behaviour.

Otherwise, you are just being cranky and argumentative. Stop it. Suck it up
and move on. The world doesn't revolve around BartC, no matter how loud he
yells.

-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From antoon.pardon at rece.vub.ac.be  Wed Dec  7 15:37:13 2016
From: antoon.pardon at rece.vub.ac.be (Antoon Pardon)
Date: Wed, 7 Dec 2016 21:37:13 +0100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29ip0$g0v$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me>
Message-ID: <801f0c6b-07b9-6adc-b229-84e7ff674810@rece.vub.ac.be>

Op 07-12-16 om 19:02 schreef BartC:
> On 07/12/2016 16:53, Michael Torrie wrote:
>> On 12/07/2016 08:48 AM, BartC wrote:
>>> I would prefer that the program "t" can be invoked exactly the same way
>>> under both systems. I don't want different instructions for Linux, or
>>> for the user (of my language) to have to write two lots of code, as that
>>> is my job...
>>
>> Ahh now we come to the crux of your argument.  You want all potential
>> platforms to conform to your own idea of what is normal.  And indeed
>> this is the actual issue that started the original thread.  But that's
>> not the way it works for any platform.  I develop a lot of code on Linux
>> that I'd like to get running on Windows.  I prefer the Linux way of
>> doing things but I'm not going to make any headway if I just try to
>> brow-beat Windows and Windows' users over the things that aren't
>> implement the same way.
> 
> There is a lack of balance.
> 
> If I run a Linux program on Windows, then the application gets to see parameters such as *.* and can expand them if it wants.
> 
> If I want to run a Windows program on Linux, and that program needs to see *.* unexpanded, then it can't undo that expansion. 
> The cat is already out of the bag.

You should stop trying to frame this as a need of the program. If it was just about the program needing to see *.* as an
argument. That is possible. Just quote the argument.

But this isn't about what a program would need, this is about how you would like things to work. You have some points I
agree with, but I don't think those points weight heavy enough to get people changing things on the unix side.

> It seems the only choice, if someone wants a cross-platform application that is invoked in the same way, is to dumb it
> down and make it assume that any parameters such as A* have been expanded, or will be expanded by that extra step.

cross-platform applications will always need to come to some kind of compromise.

> Which means that input of A B* *C will all end up running together so that you have no idea what is what or which file
> corresponds to which expansion. That's assuming these parameters have any connection to the current set of files at all; if not, then
> the input you end up is going to be meaningless.

Yes and the point is? If you write a cross-platform application and you don't consider how the environments differ
then you might end up with something meaningless.

> Now the problem is to determine whether processed input of U V W X Y Z are intended bona fide inputs, or whether
> they just happened to result from some random associations with the files in the current directory.

No it is not. Making this your problem is just an untractable mess and it is not limited to globbing. Suppose I
have a file named -rf and want it among other files removed. I don't consider it more closely and just type:

    rm -rf ...

How is your program going to decide between me wanting to remove the file '-rf' and me wanting the options?

You don't like how it generally works on unix. Too bad for you when you have to work on a unix box.
Me I don't like how it generally works on a windows. Too bad for me when I find myself without
a choice but to work on a windows box. Life just isn't easy and fair.



From rosuav at gmail.com  Wed Dec  7 15:52:54 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 8 Dec 2016 07:52:54 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <CACL+1avqbbF1CeVat88i0ghFqW=W=xJrEbHdEGJmEFuS4KDaNg@mail.gmail.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <CAPTjJmoZ0as4S5H=wD3dg5mO8XvNHNnR1KaL5-i6P_aCCdpyDA@mail.gmail.com>
 <CACL+1avqbbF1CeVat88i0ghFqW=W=xJrEbHdEGJmEFuS4KDaNg@mail.gmail.com>
Message-ID: <CAPTjJmpyJz=CcvWuF0KNm6EBFYetWLWvteWpOt680VCMtQNNOA@mail.gmail.com>

On Thu, Dec 8, 2016 at 7:17 AM, eryk sun <eryksun at gmail.com> wrote:
> On Wed, Dec 7, 2016 at 12:39 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> Note that two of the Beauty Stone tracks include quotes as well as
>> question marks. How do you identify those? Let's say you want to play
>> one of these in VLC, and then maybe you decide that the track in
>> Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild
>> it from the one in the parent directory. How does that work on
>> Windows? If you say "it doesn't", then (a) you have taken away choice
>> on a fundamental level, (b) you have your head in the sand, and (c)
>> you still haven't solved the problem of percent signs, carets, and so
>> on, which are perfectly legal in file names, but have meaning to the
>> shell.
>
> The five wildcard characters ("<>*?) aren't allowed in the names of
> files and directories -- at least not by any Windows filesystem that
> I've used. This makes it easy for a filesystem to support globbing in
> its implementation of NtQueryDirectoryFile.

Yep - see my last sentence in the quoted paragraph.

ChrisA


From ben.usenet at bsb.me.uk  Wed Dec  7 18:25:38 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Wed, 07 Dec 2016 23:25:38 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me> <874m2fg3qy.fsf@bsb.me.uk>
 <o29k6l$l4f$1@dont-email.me>
Message-ID: <8760mve2z1.fsf@bsb.me.uk>

BartC <bc at freeuk.com> writes:

> On 07/12/2016 15:25, Ben Bacarisse wrote:
>> BartC <bc at freeuk.com> writes:
>> <snip>
>>> [...] But remember:
>>>
>>>   cp *.c
>>>
>>> There might be some irate users out there if it can't detect a simple
>>> user error like that.
>>
>> There might be.  They are ill-served by current Unix shells.  But,
>> oddly, the flow has gone the other way.  People have wanted Unix shells
>> and utilities ported to Windows, not cmd.exe and friends ported to
>> Linux.  You clearly see a gap in the provision here.  Wouldn't it be
>> better for the world if you wrote cmd.exe for Linux rather than
>> complaining about what other shells do?
>
> It's not my job to write OSes.

A shell is not an OS.  The kind you want is not a big program and if you
are not alone (as I think you almost certainly are) in wanting it, you
will get help to make it happen.  Writing a simple shell used to be an
introductory exercise in system programming classes.

But maybe it's not really much of a problem for you are your main
concern is to have jolly good whinge about it?

<snip>
> But you might have seen my objections; even putting aside bias in
> favour of one OS or another, I think many of them are valid. That fact
> Linux is getting more popular does not mean it is always better.)

It's not an OS issue -- Windows has bash and Linux does not have a
compatible cmd.exe only because no one seems to have considered it
enough of a problem to be worth solving.

But, yes, I've seen all these objections before.  The problem is not
their validity by the absence of a solution with which they can be
compared.  MS's cmd.exe is not the solution (for me).  I've never had a
problem from accidentally typing cp *.c but when I had to use Windows I
had a problem almost every day getting anything done.

<snip>
-- 
Ben.


From steve+python at pearwood.info  Wed Dec  7 18:35:33 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 08 Dec 2016 10:35:33 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o2998b$ctm$1@dont-email.me>
Message-ID: <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>

On Thu, 8 Dec 2016 02:19 am, BartC wrote:

> On 07/12/2016 14:34, Steve D'Aprano wrote:
[...]
>> I don't know why you are so hung up over the number of characters here,
>> or this bogeyman of "one million files" in a directory.
> 
> Because /you/ brought it up as a reason why 'globbing' would help when
> there are command limits?

I did?

I don't remember saying any such thing. Citation required.



[...]
>> But... how does your program distinguish between the file spec *.* which
>> should be expanded, and the literal file *.* which shouldn't be expanded?
> 
> Do you mean a *.* parameter that refers to all the files in the
> directory (whether the intention is to build a list of those files or
> not) and a *.* parameter that means something entirely different?

We're talking Linux here, where "*.*" is a legal (if stupid) file name.
Existing Linux programs (mostly?) don't do their own globbing, so if I want
to list such a file, I just need to protect the metacharacters from the
shell:

[steve at ando ~]$ ls -l "*.*"
-rw-rw-r-- 1 steve steve 7 Dec  8 09:30 *.*

and there it is. Now, you *insist* that the one and only correct way to do
this is to disable shell globbing, and have each and every program do its
own globbing. Okay, I can do the first part, and because ls doesn't in
reality do its own globbing, it works fine:

[steve at ando ~]$ set -o noglob
[steve at ando ~]$ ls -l *.*
-rw-rw-r-- 1 steve steve 7 Dec  8 09:30 *.*


But now imagine that you are the author of ls, and that following your
strategy of having programs do their own globbing, ls expands that as a
glob. The shell is no longer involved here, but globbing still needs to be
defeated, only this time I'm at the mercy of each individual program, not
just a single shell.

So my question is, in your programs that you write, where you implement your
own globbing, do you support escaping metacharacters? If you were the
author of Linux ls, could I write 

    ls -l \*.\*

or possibly 

    ls -l "*.*"

or even:

    ls -l --noglob *.*

to get a listing of the file named "*.*"?

In the Unix world, I'm at the mercy of *one* program, the shell. In your
ideal world, I'm at the mercy of EVERY program that implements globbing:
each and every one has to independently offer a way to disable their own
internal globbing. How is that an improvement?

Hence my rhetorical question. When you write a program that supports
globbing, do you always remember to provide an escape mechanism?


The string quoting, escaping and expansion rules for bash are pretty darn
complicated. You have parameter expansion, aliases, two distinct types of
quoting, backslashes, shell variables, environment variables, and more,
which are applied in a certain order. Once you get past the simple cases,
bash programming is hairy and frankly scarily complex.

But at least the rules are consistent since they're applied by the shell. I
don't have to learn a separate set of rules for each and every program I
use:

"ls supports globbing, but not brace expansion..."

"rm supports globbing and brace expansion, but not parameter expansion..."

"mkdir supports tilde expansion, but incompletely..."

"touch supports only a subset of arithmetic expansion, but with bugs..."


> OK, but you go first: how does a Linux program tell the difference
> between:
> 
>    A B C
> 
> which signifies three files, and:
> 
>    A B C
> 
> which means two files, and an option C? And without using escapes!

First of all, as you well know, the convention is for options to start with
either a single or double hyphen, or sometimes a plus, and usually to come
first. So I would write:

    program -C A B


But let's suppose that this particular program doesn't follow that standard
convention, and instead has the argument signature:

    argument1 argument2 option

Fine. Where's the difficulty? The program would read the command line,
collect the first argument as `argument1`, the second argument as
`argument2`, and the third argument as `option`.

If I was particularly awful at designing user interfaces, I could even have
the signature where the third parameter could be either:

    argument1 argument2 option_or_argument3


# Python syntax
if option_or_argument3 in ("off", "on"):
    option = option_or_argument3
    argument3 = ""
else:
    option = "off"
    argument3 = option_or_argument3


I don't see what point you think you are making. How the program interprets
its command arguments is independent of the shell.


>> You need an escape mechanism too. You've done that, of course. Right?
> 
> My program wouldn't need anything so crude. The input syntax would be
> more carefully designed so as to not have the ambiguity.

Oh of course it would. How silly of me to even ask.

*rolls eyes*

I'll take that as a "No".


> And actually I generally avoid command line interfaces (ones used by
> shells) because they are such a pain. (Why, for example, aren't the
> contents of the command line - even after preprocessing by the shell -
> just read using exactly the same mechanism as subsequent lines of input?)

Because then you couldn't tell the difference between command line arguments
and standard input.

Because not all programs even need to care about reading standard input.



>>>    list fooba?.jpg {here,there}/*.jpg  another/place/*.{png,jpg} \
>>>      [a-z]/IMG* > files
>>>
>>>    command @files
>>
>> *Requiring* this would be completely unacceptable. Forcing the user to
>> write two lines (one to create a variable, then to use it) every time
>> they needed a glob expansion would go down among sys admins like a lead
>> balloon.
> 
> Suppose there is a pattern that can't be expressed as a one-liner?


You mean like this?

[steve at ando ~]$ export manyliner="hello
> world
> "
[steve at ando ~]$ echo $manyliner
hello world


But if you mean embedding newlines in the variable, well, bash is not
well-suited for that. It's possible, but a PITA. Look it up: I'm sure it
will have been answered on StackOverflow.

But what of it? What's your point? We all acknowledge that bash is not a
great choice for general purpose application programming, its a scripting
language optimized for certain tasks at the expense of others. In the areas
it has been designed for, your question doesn't come up often enough that
people care about it.


> And the shell is supposed to be a full-fledged programming language,
> people keep saying. What can be more natural than to make use of
> variables? As you go on to demonstrate!

You wouldn't write a low-level device driver in Perl, and you wouldn't write
a file utility in APL. All languages have their strengths and weaknesses.
What's your point?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Wed Dec  7 19:09:41 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 08 Dec 2016 11:09:41 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
Message-ID: <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Thu, 8 Dec 2016 02:48 am, BartC wrote:

> On 07/12/2016 13:58, Dennis Lee Bieber wrote:
>> On Wed, 7 Dec 2016 11:54:35 +0000, BartC <bc at freeuk.com> declaimed the
>> following:
>>
>>> With automatic expansion, then EVERY program can become dangerous. Can
>>> no one else see this?
>>>
>>
>> With your scheme, EVERY PROGRAM HAS TO IMPLEMENT ITS OWN GLOBBING.
> 
> Only the tiny minority that can be meaningfully invoked on an arbitrary
> number of files at the same time.


Its not just filename globbing. There's tilde expansion, brace expansion,
arithmetic expansion, parameter expansion, variable expansion, and I dare
say I've forgotten some others.

[steve at ando ~]$ echo ~
/home/steve
[steve at ando ~]$ echo $((1+7*5))
36
[steve at ando ~]$ echo $http_proxy
http://ando:3128


The echo command doesn't have to understand anything about tildes, contain
its own arithmetic expression evaluator, or even be able to parse variable
names. The shell does all that for it.

Filename globbing was just a simple example that would be familiar to
everyone discussing the topic. But the full range of expansions performed
is much, much richer than that.

[...]
> You make it sound like a big deal. Here's a program (in my language not
> Python) which prints the list of files matching a file-spec:
> 
> println dirlist(cmdparams[2])

Does dirlist provide an escaping mechanism so that you can disable filename
globbing? In another post, you claimed that in your programs, you wouldn't
use anything as clumsy and ambiguous as globbing.

    My program wouldn't need anything so crude. The input syntax 
    would be more carefully designed so as to not have the ambiguity.

So presumably your dirlist() command can distinguish between the file called
literally "*.*" and the file spec "*.*" that should be expanded, only of
course you don't actually use globbing syntax, as that's too crude, you use
your own ever-so-clever syntax which is completely ambiguity free.

Right?

And of course your program is also capable of variable and arithmetic
expansion, right?

[steve at ando ~]$ export base="thefile"
[steve at ando ~]$ ls -l "$base$((1000 + 234))"
-rw-rw-r-- 1 steve steve 0 Dec  8 10:51 thefile1234

 
> (Here are implementations of that dirlist() for both OSes:
> http://pastebin.com/knciRULE

I have no idea how good the Windows globbing support is, or whether it can
be escaped. (Presumably its unambiguous because * ? [ ] are not legal in
Windows file names?)

For the Linux implementation, I don't see how extractpath and extractfile
are defined, or fnmatch, so who knows how many bugs are in your globbing
implementation.

I see no obvious way of disabling globs across the entire command line,
equivalent to the bash "setopt -o noglob".

I see no brace expansion.

I see no arithmetic expansion.

I see no variable expansion.

I don't even see partial support for ~ expansion, let alone full support.

Looks like you have a lot of wheels that need re-inventing before you come
even close to parity with the features of the Linux shell.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Wed Dec  7 19:16:50 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 08 Dec 2016 11:16:50 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <28877D50-0E7D-473D-AC44-6FBE99712F3E@gmail.com>
 <CANc-5Ux6MLfQ++pMGtYC5MqSCQqr=N-cvTyO0sFh8vhYV4DMLA@mail.gmail.com>
 <mailman.142.1481140873.2253.python-list@python.org>
Message-ID: <5848a674$0$1598$c3e8da3$5496439d@news.astraweb.com>

On Thu, 8 Dec 2016 07:01 am, Skip Montanaro wrote:

> Some things are better just left alone. File globbing is probably one
> of those things. There's no right answer, and the people in the two
> camps will never come to a compromise.

Be fair Skip, we've already repeatedly acknowledged that the Unix model of
shell expansion doesn't suit all use-cases for a shell. As have the
designers of the shells, as they typically provide ways of turning off
expansions.

Its only Bart who insists that his way is always the best way, and that 40
years of Unix sys admins who have designed the shell to work the way they
want are wrong.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From bc at freeuk.com  Wed Dec  7 19:34:01 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 00:34:01 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o2998b$ctm$1@dont-email.me>
 <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2a9nc$3en$1@dont-email.me>

On 07/12/2016 23:35, Steve D'Aprano wrote:
> On Thu, 8 Dec 2016 02:19 am, BartC wrote:
>
>> On 07/12/2016 14:34, Steve D'Aprano wrote:
> [...]
>>> I don't know why you are so hung up over the number of characters here,
>>> or this bogeyman of "one million files" in a directory.
>>
>> Because /you/ brought it up as a reason why 'globbing' would help when
>> there are command limits?
>
> I did?
>
> I don't remember saying any such thing. Citation required.

In the part you snipped:

On 07/12/2016 05:15, Steven D'Aprano wrote:
 > On Wednesday 07 December 2016 12:55, BartC wrote:
 >
 >> But even Linux's 128KB will fill [fail?] if someone wanted a command 
line that
 >> listed 20,000 files individually. But it would be spectacularly bad use
 >> of a command line interface which was designed for humans.
 >
 > That's EXACTLY the point of having the shell do globbing.

-- 
Bartc


From rosuav at gmail.com  Wed Dec  7 19:52:22 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 8 Dec 2016 11:52:22 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o2998b$ctm$1@dont-email.me>
 <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmrmzcBhRBP2SCaRusq+6b31_wvWJtg++Hsw_foaxdUJ+Q@mail.gmail.com>

On Thu, Dec 8, 2016 at 10:35 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> In the Unix world, I'm at the mercy of *one* program, the shell. In your
> ideal world, I'm at the mercy of EVERY program that implements globbing:
> each and every one has to independently offer a way to disable their own
> internal globbing. How is that an improvement?

Moreover, you're at the mercy of the one program *you chose* out of a
set of similar options (shells). There are some times when bash's
style of tab completion feels a bit clunky, and I'd really like the
"show a few options and let you pick with the arrow keys" style of...
is it zsh? And I have the freedom to choose that if I want to, without
reference to the program being invoked. Try doing THAT with the
Windows model. Can you make it so that, instead of expanding the whole
glob, you pop up something that lets you multiselect from within that
pattern? You type "cp *.c dest/" and it opens up a thing listing all
.c files and lets you choose which to copy? I don't know if there are
any shells like that, but *one could be written*, and it would
instantly apply not just to the copy command, but to every other
command you might want to use. Want to make symlinks to a whole bunch
of things? The ln command accepts multiple targets and a destination
directory. In Windows, you'd have to add this feature to every program
individually, or (more likely) produce a single overarching file
management program that has all these features. Oh, but then it
doesn't work if you want to invoke VLC on all those files, because the
programmer didn't think of that. Whoops.

ChrisA


From bc at freeuk.com  Wed Dec  7 20:15:58 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 01:15:58 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2ac62$9vk$1@dont-email.me>

On 08/12/2016 00:09, Steve D'Aprano wrote:
> On Thu, 8 Dec 2016 02:48 am, BartC wrote:

>> You make it sound like a big deal. Here's a program (in my language not
>> Python) which prints the list of files matching a file-spec:
>>
>> println dirlist(cmdparams[2])
>
> Does dirlist provide an escaping mechanism so that you can disable filename
> globbing?

No. It simply produces a list of files in a path matching a particular 
wildcard pattern using * and ?.

That's all. I know the value of keeping things straightforward instead 
of throwing in everything you can think of. The file-matching is done by 
WinAPI functions. The Linux version was done last week and I stopped 
when I proved the concept (of having a dirlist() that worked - to 
certain specs - on both OSes, 32- and 64-bit, without change).

  In another post, you claimed that in your programs, you wouldn't
> use anything as clumsy and ambiguous as globbing.

>     My program wouldn't need anything so crude. The input syntax
>     would be more carefully designed so as to not have the ambiguity.

I meant designing a CLI where *.* could be used in different parameters 
to mean different things.

Reading that post again, you presumably meant either *.* used to match 
all files (with an embedded "." in Linux, with or without in Windows; 
another difference), or *.* used to match a specific file called *.*.


> So presumably your dirlist() command can distinguish between the file called
> literally "*.*" and the file spec "*.*" that should be expanded,

No. I won't support that (not unless it's supported by Posix's 
fnmatch()). Because it's the thin end of the wedge. I can show a few 
lines of code and then you will say, Ah, but what about this...

And I first used this function in early 90s I think, I don't recall it 
ever not working.

> And of course your program is also capable of variable and arithmetic
> expansion, right?

Um, dirlist() is used within a language, of course it can do all that. 
If you mean within the file-pattern string submitted to dirlist, then I 
don't even know what that would mean.


>> (Here are implementations of that dirlist() for both OSes:
>> http://pastebin.com/knciRULE

[Sorry about that name; I didn't choose it!]

> I have no idea how good the Windows globbing support is, or whether it can
> be escaped.

Why such a preoccupation with 'globbing'? It's something I may use from 
time to time with DIR or COPY or something, and that's it.

I just didn't know someone could also use it with any user program and 
that program's command line would then fill with loads of unexpected 
files that can mess things up.

  (Presumably its unambiguous because * ? [ ] are not legal in
> Windows file names?)
>
> For the Linux implementation, I don't see how extractpath and extractfile
> are defined, or fnmatch,

man fnmatch


> Looks like you have a lot of wheels that need re-inventing before you come
> even close to parity with the features of the Linux shell.

These are wheels I don't *want* to re-invent! I'm not writing a shell.

I suspect we might be talking at cross-purposes though.

-- 
Bartc


From steve+comp.lang.python at pearwood.info  Wed Dec  7 20:54:40 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 08 Dec 2016 12:54:40 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <o2998b$ctm$1@dont-email.me>
 <58489cc6$0$1620$c3e8da3$5496439d@news.astraweb.com>
 <o2a9nc$3en$1@dont-email.me>
Message-ID: <5848bd62$0$1530$c3e8da3$5496439d@news.astraweb.com>

On Thursday 08 December 2016 11:34, BartC wrote:

> On 07/12/2016 23:35, Steve D'Aprano wrote:
>> On Thu, 8 Dec 2016 02:19 am, BartC wrote:
>>
>>> On 07/12/2016 14:34, Steve D'Aprano wrote:
>> [...]
>>>> I don't know why you are so hung up over the number of characters here,
>>>> or this bogeyman of "one million files" in a directory.
>>>
>>> Because /you/ brought it up as a reason why 'globbing' would help when
>>> there are command limits?
>>
>> I did?
>>
>> I don't remember saying any such thing. Citation required.
> 
> In the part you snipped:
> 
> On 07/12/2016 05:15, Steven D'Aprano wrote:
>  > On Wednesday 07 December 2016 12:55, BartC wrote:
>  >
>  >> But even Linux's 128KB will fill [fail?] if someone wanted a command
> line that
>  >> listed 20,000 files individually. But it would be spectacularly bad use
>  >> of a command line interface which was designed for humans.
>  >
>  > That's EXACTLY the point of having the shell do globbing.

Right. I meant that rather than have to list 20,000 files by hand, which would 
be a "spectacularly bad use of a command line interface", I can use globbing.

I think in this particular question we're just responding to different things. 
I didn't mean that globbing was a solution to the limits of command line 
length. I meant it was a solution to the poor command line interface (i.e. 
expecting the user to write out all 20,000 files by hand).

In fact, I think I was the first person to mention the command line limits, and 
linked to a page that discussed them in detail.

(The "Too Many Arguments" limit that people occasionally run into isn't a shell 
limitation, but the Unix exec command limit.)



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From torriem at gmail.com  Wed Dec  7 21:05:55 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 19:05:55 -0700
Subject: PonyORM: generators as a query syntax
Message-ID: <2136a29c-d356-81e2-16e4-7d0b3745afdf@gmail.com>

I was just made aware of a very interesting ORM project that has been
around since about 2013, while listening to a recent episode of the Talk
Python To Me podcast.  The idea of using generators to build queries is
really cool.  I'm sure PonyORM has its limitations and drawbacks, as all
ORM models do.  But I like what see so far, particularly how simple it
is from a programmer's point of view.  Thought others may not have heard
of it either, but would find it interesting:

https://ponyorm.com/

Anyone have any real-world experience with it vs some other ORM like
SQAlchemy?  I have a small database project that I may use PonyORM with
so we'll see how it goes.  I have not yet determined which versions of
Python it works with. Hopefully it supports Python 3.

I have no idea what kind of black magic is used under the hood to build
the SQL queries.


From steve+comp.lang.python at pearwood.info  Wed Dec  7 22:41:00 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 08 Dec 2016 14:41 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
Message-ID: <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>

On Thursday 08 December 2016 12:15, BartC wrote:

> On 08/12/2016 00:09, Steve D'Aprano wrote:
>> On Thu, 8 Dec 2016 02:48 am, BartC wrote:
> 
>>> You make it sound like a big deal. Here's a program (in my language not
>>> Python) which prints the list of files matching a file-spec:
>>>
>>> println dirlist(cmdparams[2])
>>
>> Does dirlist provide an escaping mechanism so that you can disable filename
>> globbing?
> 
> No. It simply produces a list of files in a path matching a particular
> wildcard pattern using * and ?.
> 
> That's all. I know the value of keeping things straightforward instead
> of throwing in everything you can think of. The file-matching is done by
> WinAPI functions. The Linux version was done last week and I stopped
> when I proved the concept (of having a dirlist() that worked - to
> certain specs - on both OSes, 32- and 64-bit, without change).

So you're happy with the fact that there are legitimate file names that your 
program simply has no way of dealing with?

Thank you for making my point for me! If you leave the implementation of 
metacharacter expansion up to the individual program, then you get the 
situation where individual programs will fail to support certain features, or 
even basic functionality.

Python's fnmatch lib is a good example. It has, or at least had, no support for 
escaping metacharacters. Anyone relying on Python's fnmatch and glob modules 
alone for globbing will be unable to handle legitimate file names.



>   In another post, you claimed that in your programs, you wouldn't
>> use anything as clumsy and ambiguous as globbing.
> 
>>     My program wouldn't need anything so crude. The input syntax
>>     would be more carefully designed so as to not have the ambiguity.
> 
> I meant designing a CLI where *.* could be used in different parameters
> to mean different things.
> 
> Reading that post again, you presumably meant either *.* used to match
> all files (with an embedded "." in Linux, with or without in Windows;
> another difference), or *.* used to match a specific file called *.*.

Correct.

It doesn't matter whether metacharacters are expanded by the shell or the 
program itself, there needs to be an escaping mechanism for those cases where 
you want the metacharacters to be treated literally.



>> So presumably your dirlist() command can distinguish between the file called
>> literally "*.*" and the file spec "*.*" that should be expanded,
> 
> No. I won't support that (not unless it's supported by Posix's
> fnmatch()). Because it's the thin end of the wedge. I can show a few
> lines of code and then you will say, Ah, but what about this...
> 
> And I first used this function in early 90s I think, I don't recall it
> ever not working.

If you can't use it to specify a file called literally "*.*", then its not 
working.


>> And of course your program is also capable of variable and arithmetic
>> expansion, right?
> 
> Um, dirlist() is used within a language, of course it can do all that.
> If you mean within the file-pattern string submitted to dirlist, then I
> don't even know what that would mean.

I showed you an example in another post:

[steve at ando ~]$ export base="thefile"
[steve at ando ~]$ ls -l "$base$((1000 + 234))"
-rw-rw-r-- 1 steve steve 0 Dec  8 10:51 thefile1234


Of course it is silly to write 1000 + 234 as a literal constant like that, but 
either or both of those could just as easily be variables.

The point here is not that *you* should build a calculator into your version of 
ls. The complete opposite: the point is that application authors don't have to 
build in more and more non-core functionality into their own applications, 
because the shell handles all this peripheral functionality for them.

The user then can choose whether to use the shell's extra functionality or not, 
and your application doesn't need to care one way or the other.



>>> (Here are implementations of that dirlist() for both OSes:
>>> http://pastebin.com/knciRULE
> 
> [Sorry about that name; I didn't choose it!]
> 
>> I have no idea how good the Windows globbing support is, or whether it can
>> be escaped.
> 
> Why such a preoccupation with 'globbing'? It's something I may use from
> time to time with DIR or COPY or something, and that's it.

Right -- and that's why you're not the target audience for Unix shells. Unix 
sys admins use globs and other shell expansion features extensively, probably 
hundreds of times a day. Even me, hardly a sys admin at all, use it dozens of 
times a day.

For example, a quick and simple way of backing up a file with a date stamp:

steve at runes:~$ cp alloc.txt{,-`date +%Y%m%d`}
steve at runes:~$ ls alloc.txt* 
alloc.txt  alloc.txt-20161208


> I just didn't know someone could also use it with any user program and
> that program's command line would then fill with loads of unexpected
> files that can mess things up.

You're treating this as some unknown third party dumping junk into the command 
line. Its not. Its me, the user, *chosing* to use another program (the shell) 
to expand a bunch of metacharacters, to save me having to type things out by 
hand.

You're worried that 

    program *

will expand to a million file names? Okay, but what if I, the user, typed out a 
million file names by hand? I'd hate to do that, but if I need to process a 
million file names, by Jove that's what I'll do. (With tab completion, it won't 
be *quite* as awful as it seems.)

Or... I'll write a macro or script to generate a million file names, and pass 
them to your program.

Or... I'll make use of an existing program, the shell, and use that. Because 
I'm not an idiot and there's no way on Earth I'm actually going to type out a 
million file names, not when I have a computer that will do it for me.

So what's the difference? Why should you prohibit me from using automation to 
make my life easier?

Bart, we get it that you were taken by surprise by shell expansion, because in 
X decades of professional IT work you'd never seen it before. Okay, you were 
surprised. Oops, sorry. That's the risk you take when you move from a platform 
you know and are comfortable with onto an unfamiliar platform that breaks your 
expectations, and I do sympathise with your sense of shock.

But it's not a bug, its a feature, and Linux system admins use that feature, 
and others like it, *extensively*. Its not a feature you care for, but its not 
designed for you.

So get over it.

[...]
>> Looks like you have a lot of wheels that need re-inventing before you come
>> even close to parity with the features of the Linux shell.
> 
> These are wheels I don't *want* to re-invent! I'm not writing a shell.

Indeed, and nobody wants to force you to.

But the people who wrote sh, bash, zsh, etc ARE writing shells, and they've 
given them the features that shell users demand so that you, the application 
author, don't have to.




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Thu Dec  8 00:16:50 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 08 Dec 2016 16:16:50 +1100
Subject: Get min and max dates
References: <o29cgh$nlj$3@dont-email.me>
Message-ID: <5848ecc5$0$1605$c3e8da3$5496439d@news.astraweb.com>

On Thursday 08 December 2016 03:15, DFS wrote:

> dts=  ['10-Mar-1998',
>         '20-Aug-1997',
>         '06-Sep-2009',
>         '23-Jan-2010',
>         '12-Feb-2010',
>         '05-Nov-2010',
>         '03-Sep-2009',
>         '07-Nov-2014',
>         '08-Mar-2013']
> 
> Of course, the naive:
> min(dates) = '03-Sep-2009'
> max(dates) = '23-Jan-2010'
> is wrong.
> 
> Not wanting to use any date parsing libraries, I came up with:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That's where you have gone wrong. By not using a date parsing library which 
works and has been tested, you have to write your own dodgy and possibly buggy 
parsing routine yourself.

Why reinvent the wheel?


> ========================================================================
> m=[('Dec','12'),('Nov','11'),('Oct','10'),('Sep','09'),
>     ('Aug','08'),('Jul','07'),('Jun','06'),('May','05'),
>     ('Apr','04'),('Mar','03'),('Feb','02'),('Jan','01')]
> 
> #create new list with properly sortable date (YYYYMMDD)
> dts2 = []
> for d in dts:
> dts2.append((d[-4:]+dict(m)[d[3:6]]+d[:2],d))


Why do you convert m into a dict each and every time through the loop? If you 
have a million items, you convert m to a dict a million times, and then throw 
away the dict a million times. And then you'll complain that Python is slow...

m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
     'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}

There you go, fixed that.

As for the rest, I haven't bothered to check your code. But here's a much 
simpler, more Pythonic, way:

def date_to_seconds(string):
    return time.mktime(time.strptime(string, '%d-%b-%Y'))


min(dts, key=date_to_seconds)  # returns '20-Aug-1997'
max(dts, key=date_to_seconds)  # returns '07-Nov-2014'


Works for at least Python 3.3 or better.

The best part of this is that you can now support any time format you like, 
just by changing the format string '%d-%b-%Y'.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Thu Dec  8 00:27:41 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 08 Dec 2016 16:27:41 +1100
Subject: sets anomaly
References: <526563a5-4e1f-4098-a75f-d65282b4ef77@googlegroups.com>
Message-ID: <5848ef4f$0$11121$c3e8da3@news.astraweb.com>

On Thursday 08 December 2016 02:17, Rustom Mody wrote:

> Trying to write some code using sets (well frozen sets)
> And was hit by this anomaly
> 
> This is the behavior of lists I analogously expect in sets:
> 
>>>> []
> []
>>>> [[]]
> [[]]
>>>> 
> 
> ie the empty list and the list of the empty list are different things

That's a property of the *list display*, [], not the list() constructor. The 
list constructor takes an iterable of items, so if you pass it an empty 
iterable, you get an empty list:

py> list([])
[]

Since there's no frozenset display, there's no analogy with [[]], and you 
similarly get a single empty frozen set:

py> frozenset([])
frozenset()

Notice the repr()? Like list(), tuple(), set() and dict(), calling frozenset() 
with no arguments returns an empty frozenset:

py> frozenset()
frozenset()


> And then some figuring out how to get an empty set into a set
> This is the best I get:
>>>> f([f([])])
> frozenset({frozenset()})


py> ? = frozenset()
py> frozenset([?])
frozenset({frozenset()})



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From greg.ewing at canterbury.ac.nz  Thu Dec  8 01:03:02 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 19:03:02 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <o28usn$89t$1@dont-email.me>
 <58481de0$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <easbcoFe90nU1@mid.individual.net>

Steve D'Aprano wrote:
> I don't know any Unix programs that provide
> file spec processing.

'find' does, but only for its -name argument, and only because
it does something different with it from what the shell would
do. And you do need to quote it if it contains glob characters.

-- 
Greg


From plzr at mac.com  Thu Dec  8 01:09:42 2016
From: plzr at mac.com (3dB)
Date: Wed, 7 Dec 2016 22:09:42 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
Message-ID: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>

trying to install SpeechRecognition for Python results in error:

running install_lib
    creating /Library/Python/2.7/site-packages/speech_recognition
    error: could not create '/Library/Python/2.7/site-packages/speech_recognition': Permission denied


Any advice on how to fix?

Follow information may be helpful:

MacOSX 10.12.1
Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin

/usr/local/bin/brew
/usr/bin/python

pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)



From greg.ewing at canterbury.ac.nz  Thu Dec  8 01:10:11 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 19:10:11 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <384e999d-6220-4b27-83ee-07c27de5afe8@googlegroups.com>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <b7CdnWUZicc399vFnZ2dnUU7-bnNnZ2d@giganews.com>
 <o266t3$bnd$1@dont-email.me> <eanpt6FbbkcU1@mid.individual.net>
 <o26e5l$4k6$1@dont-email.me>
 <CAPTjJmoZ1SPunjBju2rRfiSU+jFUc5=CFaivRRNv6uAEd6pXcQ@mail.gmail.com>
 <mailman.83.1481031282.2253.python-list@python.org>
 <o26fo7$a7s$1@dont-email.me> <eaopp7Fj5v8U1@mid.individual.net>
 <384e999d-6220-4b27-83ee-07c27de5afe8@googlegroups.com>
Message-ID: <easbq5FebpoU1@mid.individual.net>

Paul Moore wrote:
> On Tuesday, 6 December 2016 21:44:18 UTC, Gregory Ewing  wrote:
> 
>> What you need to understand is that, to a Unix user, * and ? are *just as
>> well known* as |, < and >.
> 
> And to Windows users. Just because BartC is proposing a silly distinction
> between what "Unix users" and "Windows users" expect, doesn't mean there is
> such a distinction.

Yes, Windows users will know about * too, but their understanding
of it will be different.

The real point is that Unix users will know that it has a
special meaning to the shell and work accordingly. They won't
be tripped up by the kind of problems Bart imagines they
will have with arguments containing *.

-- 
Greg


From torriem at gmail.com  Thu Dec  8 01:18:42 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 7 Dec 2016 23:18:42 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
Message-ID: <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>

On 12/07/2016 11:09 PM, 3dB wrote:
> trying to install SpeechRecognition for Python results in error:
> 
> running install_lib
>     creating /Library/Python/2.7/site-packages/speech_recognition
>     error: could not create '/Library/Python/2.7/site-packages/speech_recognition': Permission denied
> 
> 
> Any advice on how to fix?

Are you using sudo?

> 
> Follow information may be helpful:
> 
> MacOSX 10.12.1
> Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
> 
> /usr/local/bin/brew
> /usr/bin/python
> 
> pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
> 



From greg.ewing at canterbury.ac.nz  Thu Dec  8 01:20:01 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 19:20:01 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29au2$ip2$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
Message-ID: <easccjFefl8U1@mid.individual.net>

BartC wrote:
> And globbing doesn't take care of all of it: a Linux program still has 
> to iterate over a loop of filenames. The same as on Windows, except the 
> latter will need to call a function to deliver the next filename.

Actually, most of them will require *two* loops, one to
iterate over a sequence of filespecs, and another for
the files matching each filespec.

Whereas the unix program only requires one loop.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Dec  8 01:23:04 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 19:23:04 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29au2$ip2$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
Message-ID: <easciaFeh88U1@mid.individual.net>

BartC wrote:
> Only the tiny minority that can be meaningfully invoked on an arbitrary 
> number of files at the same time.

Um... you mean the "tiny minority" that includes just
about *every* unix utility that operates on files?

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Dec  8 01:26:01 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 19:26:01 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29au2$ip2$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
Message-ID: <eascnrFeialU1@mid.individual.net>

BartC wrote:
>  println dirlist(cmdparams[2])        # Windows
>  println tail(cmdparams)              # Linux

I'm not familiar with your language, so I'll reply
in Python. If you write it like this:

    for arg in sys.argv[1:]:
       for name in glob(arg):
          print(name)

then the user will be able to invoke it the same way
on both systems.

-- 
Greg


From torriem at gmail.com  Thu Dec  8 02:06:43 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 00:06:43 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <eascnrFeialU1@mid.individual.net>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <eascnrFeialU1@mid.individual.net>
Message-ID: <3ab1a823-c32f-dd10-707f-7f0398ff3a4d@gmail.com>

On 12/07/2016 11:26 PM, Gregory Ewing wrote:
> BartC wrote:
>>  println dirlist(cmdparams[2])        # Windows
>>  println tail(cmdparams)              # Linux
> 
> I'm not familiar with your language, so I'll reply
> in Python. If you write it like this:
> 
>     for arg in sys.argv[1:]:
>        for name in glob(arg):
>           print(name)
> 
> then the user will be able to invoke it the same way
> on both systems.

With the caveat that certain filenames may cause issues on Linux.  For
example, a file like "Chicago - Will You Still Love Me?.ogg".  Or a
filename with a "*" in the name.  Probably it would still work since
each of these filenames when taken as a glob actually match themselves
when expanded.  But they could match other files too that the user did
not want to pass in.

Personally I'd include logic to only run args through glob if I was on
Windows.

These are the perils of cross-platform compatibility.



From greg.ewing at canterbury.ac.nz  Thu Dec  8 02:24:30 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 08 Dec 2016 20:24:30 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29ip0$g0v$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me>
Message-ID: <easg5gFfa6vU1@mid.individual.net>

BartC wrote:
> Which means that input of A B* *C will all end up running together so 
> that you have no idea what is what or which file corresponds to which 
> expansion.

So if you intend your program to be used on unix, you
need to design it so that it doesn't depend on such
distinctions.

> Now the problem is to determine whether processed input of U V W X Y Z 
> are intended bona fide inputs, or whether they just happened to result 
> from some random associations with the files in the current directory.

You're entitled to assume that a Unix user using your
program from the shell is reasonably intelligent and at
least moderately familiar with how Unix shells behave.

Such a user will *not* be using unescaped wildcards
unless they *intend* them to be expanded. So you are
not going to get "random associations with the files
in the current directory". That's another imaginary
bogeyman.

> But this is what happens when you have to work to the lowest common 
> denominator.

If you're unwilling to do things differently on different
platforms, then yes, you need to restrict yourself to
their intersection. That's an inescapable fact of logic.

> (I'm in the middle of porting my console editor to Linux. But one 
> problem is that on one Linux, half the key combinations (eg. 
> Shift+Ctrl+B) are not recognised.

If you're reading characters from a tty device in raw
mode (which I assume is what you mean by "console editor")
I'm not aware of *any* Unix system that will let you
distinguish between Ctrl+B and Shift+Ctrl+B that way.
That's because the tty driver delivers ASCII characters,
and there are no separate ASCII codes for shifted control
characters.

> Except that was only two Linuxes; perhaps on others, the keyboard will 
> likely be crippled in some other way.

No, they'll all be the same -- if it has an ASCII code,
you'll be able to get it from a tty device, otherwise you
won't.

> How people manage to do anything on such an OS I've no idea.

Programs that need to be able to distinguish all of the
modifiers are normally implemented as GUI applications,
which get keyboard input a different way.

-- 
Greg


From amamaenko at gmail.com  Thu Dec  8 04:15:03 2016
From: amamaenko at gmail.com (Anton Mamaenko)
Date: Thu, 8 Dec 2016 12:15:03 +0300
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
Message-ID: <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>

I would suggest using virtual environment (virtualenv, for example) for installing such packages. 

Dealing with directory permissions on MacOS is complicated, and using "sudo" is not the right way. Moreover, during the next OS upgrade the permissions will be updated as well, and chances are that your global package configuration breaks anyways. 

Regards,
      Anton

>> On 8 Dec 2016, at 09:18, Michael Torrie <torriem at gmail.com> wrote:
>> 
>> On 12/07/2016 11:09 PM, 3dB wrote:
>> trying to install SpeechRecognition for Python results in error:
>> 
>> running install_lib
>>   creating /Library/Python/2.7/site-packages/speech_recognition
>>   error: could not create '/Library/Python/2.7/site-packages/speech_recognition': Permission denied
>> 
>> 
>> Any advice on how to fix?
> 
> Are you using sudo?
> 
>> 
>> Follow information may be helpful:
>> 
>> MacOSX 10.12.1
>> Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
>> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
>> 
>> /usr/local/bin/brew
>> /usr/bin/python
>> 
>> pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From peter.heitzer at rz.uni-regensburg.de  Thu Dec  8 04:23:49 2016
From: peter.heitzer at rz.uni-regensburg.de (Peter Heitzer)
Date: 8 Dec 2016 09:23:49 GMT
Subject: Get min and max dates
References: <o29cgh$nlj$3@dont-email.me>
Message-ID: <easn55FfvmbU3@mid.individual.net>

DFS <nospam at dfs.com> wrote:
>dts=  ['10-Mar-1998',
>       '20-Aug-1997',
>       '06-Sep-2009',
>       '23-Jan-2010',
>       '12-Feb-2010',
>       '05-Nov-2010',
>       '03-Sep-2009',
>       '07-Nov-2014',
>       '08-Mar-2013']

>Of course, the naive:
>min(dates) = '03-Sep-2009'
>max(dates) = '23-Jan-2010'
>is wrong.

>Not wanting to use any date parsing libraries, I came up with:
>========================================================================
>m=[('Dec','12'),('Nov','11'),('Oct','10'),('Sep','09'),
>   ('Aug','08'),('Jul','07'),('Jun','06'),('May','05'),
>   ('Apr','04'),('Mar','03'),('Feb','02'),('Jan','01')]

>#create new list with properly sortable date (YYYYMMDD)
>dts2 = []
>for d in dts:
>       dts2.append((d[-4:]+dict(m)[d[3:6]]+d[:2],d))

>print 'min: ' + min(dts2)[1]
>print 'max: ' + max(dts2)[1]
>========================================================================
>$python getminmax.py
>min: 20-Aug-1997
>max: 07-Nov-2014

>which is correct, but I sense a more pythonic way, or a one-liner list 
>comprehension, is in there somewhere.
I'd use strptime from the time module.

Then you could write
dts2.append(strptime(d,'%d-%b-%Y)
min and max return a struct_time type that can easily converted to
the original date format

-- 
Dipl.-Inform(FH) Peter Heitzer, peter.heitzer at rz.uni-regensburg.de


From cxielamiko at gmail.com  Thu Dec  8 05:50:31 2016
From: cxielamiko at gmail.com (Vitaly Pavlenko)
Date: Thu, 8 Dec 2016 02:50:31 -0800 (PST)
Subject: Snakify - free introductory Python online course with exercises
Message-ID: <3d1e08cd-d09e-49f2-9a1a-f1451dc85cd0@googlegroups.com>

Hi fellow Pythonistas,

Let me show you my project - an introductory Python online course: https://snakify.org/ . Our key features are 100+ exercises with deep mathematical approach and a neat website to solve them online using a step-by-step debugger. We already have thousands of happy users each month in Russia, including 30 high schools and 7 universities, and we started being used in some UK and US schools.

Please, share this link to anyone who starts learning Python. 

Thanks,
Vitaly Pavlenko
cxielamiko @gmail.com


From bc at freeuk.com  Thu Dec  8 07:43:18 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 12:43:18 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <easg5gFfa6vU1@mid.individual.net>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
Message-ID: <Jzc2A.281652$kM7.92263@fx44.am4>

On 08/12/2016 07:24, Gregory Ewing wrote:
> BartC wrote:

>> (I'm in the middle of porting my console editor to Linux. But one
>> problem is that on one Linux, half the key combinations (eg.
>> Shift+Ctrl+B) are not recognised.
>
> If you're reading characters from a tty device in raw
> mode (which I assume is what you mean by "console editor")
> I'm not aware of *any* Unix system that will let you
> distinguish between Ctrl+B and Shift+Ctrl+B that way.
> That's because the tty driver delivers ASCII characters,
> and there are no separate ASCII codes for shifted control
> characters.

Run the code below and start pressing keys. On both of my Linuxes, I get 
escape sequences shown when I Insert, Delete, Home, End, Page Up, Page 
Down, Up, Down, Left, Right and most of the function keys; not just 
single ASCII codes.

But I also get different sequences, on Ubuntu, when I press Shift, Ctrl 
or Alt with those keys, but not all shifts nor combinations will work 
(some have special meanings anyway).

Then I try the same on Debian (I think it is) on a Raspberry Pi, and 
most Shift and Ctrl are ignored, except for Ctrl A to Z (with a few gaps).

(Neither will see Shift+Ctrl+B, which means go to start of the file, 
same as Ctrl+Home. Ubuntu sees Ctrl+Home, but not Debian, although it 
will report Alt+Home. And some laptop keyboards already have Home on an 
Alternate-Function shift! It's a mess.)

>> Except that was only two Linuxes; perhaps on others, the keyboard will
>> likely be crippled in some other way.
>
> No, they'll all be the same -- if it has an ASCII code,
> you'll be able to get it from a tty device, otherwise you
> won't.
>

>> How people manage to do anything on such an OS I've no idea.
>
> Programs that need to be able to distinguish all of the
> modifiers are normally implemented as GUI applications,
> which get keyboard input a different way.

How do they work; what magic do they use to get that key information, 
and why can't it be done outside of a GUI? As I understand a Linux GUI 
is built on top of Linux.

----------------------
# Python 2 because of the 'print' handling

def getch():     # adapted from first getch I saw on the internet
         import sys, tty, termios
         fd = sys.stdin.fileno()
         old_settings = termios.tcgetattr(fd)
         tty.setraw(sys.stdin.fileno())
         ch = sys.stdin.read(1)
         termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
         return ord(ch)

print "Press keys"
print ("Hit Escape twice to quit")

escape=0
while 1:
     k=getch()

     if k==27:
         print
         print "Esc ",
         if escape: break
     elif k==13:
         print "Enter"
     elif k==10:
         print "Newline"
     elif k<=26:
         print "Ctrl",chr(k+64)
     else:
         print chr(k),

     escape = k==27

----------------------

(On another test, using a C-based getch(), pressing Enter returns code 
10 not 13; another difference to note. Either code appears to clash with 
Ctrl-M or Ctrl-J, a difference from Windows where Ctrl-J, Ctrl-M and 
Enter are distinct keys, as they are in actuality.)

-- 
Bartc


From greg.ewing at canterbury.ac.nz  Thu Dec  8 08:40:28 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 09 Dec 2016 02:40:28 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <Jzc2A.281652$kM7.92263@fx44.am4>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
Message-ID: <eat66eFkfp9U1@mid.individual.net>

BartC wrote:
> Run the code below and start pressing keys. On both of my Linuxes, I get 
> escape sequences shown when I Insert, Delete, Home, End, Page Up, Page 
> Down, Up, Down, Left, Right and most of the function keys; not just 
> single ASCII codes.

That's probably because your terminal window is emulating
some model of glass teletype and sending the same sequence
of characters that a real terminal of that kind would
send in response to those keys. You'll likely get different
sequences depending on what terminal is being emulated.

> How do they work; what magic do they use to get that key information, 
> and why can't it be done outside of a GUI?

It doesn't have to be done in a GUI, but you'd have to
deal with whatever raw keyboard input device the GUIs are
built on. I'm not familiar with the details on Linux.
Look into /dev/input if you're curious.

-- 
Greg


From marco.buttu at gmail.com  Thu Dec  8 09:51:48 2016
From: marco.buttu at gmail.com (Marco Buttu)
Date: Thu, 08 Dec 2016 15:51:48 +0100
Subject: Name resolution and the (wrong?) LEGB rule
Message-ID: <58497384.9020900@oa-cagliari.inaf.it>

Sometimes the Python name resolution is explained using a LEGB rule.
For instance, in [1] (I think also the Learning Python book gives the same):

"if a particular name:object mapping cannot be found in the local 
namespaces, the namespaces of the enclosed scope are being searched 
next. If the search in the enclosed scope is unsuccessful, too, Python 
moves on to the global namespace, and eventually, it will search the 
built-in namespace (side note: if a name cannot found in any of the 
namespaces, a NameError will is raised)."

AFAIK, Python has static scoping: the scope for a name is given during 
the bytecode compilation. This means that before executing the program, 
Python already know in which namespace to look for an object. So, it 
seems to me that the LEGB rule is wrong, and this is what actually happens:

* local name (LOAD_FAST instruction): Python looks (only) in the local 
namespace, and if it does not find the name, then it raises an 
UnboundLocalError

* global name (LOAD_GLOBAL): at first Python looks in the globals(), and 
in case the name is not there, it looks in the builtins namespace; if 
the name is neither in the global nor in the builtin namespace, it 
raises a NameError

* enclosing scope (LOAD_DEREF): there is a closure, and Python looks for 
the name in the enclosing namespace

Is that right, or am I missing something? Thanks, Marco

[1] 
http://sebastianraschka.com/Articles/2014_python_scope_and_namespaces.html

-- 
Marco Buttu

INAF-Osservatorio Astronomico di Cagliari
Via della Scienza n. 5, 09047 Selargius (CA)
Phone: 070 711 80 217
Email: mbuttu at oa-cagliari.inaf.it



From random832 at fastmail.com  Thu Dec  8 10:37:27 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 08 Dec 2016 10:37:27 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <easccjFefl8U1@mid.individual.net>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
Message-ID: <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>

On Thu, Dec 8, 2016, at 01:20, Gregory Ewing wrote:
> BartC wrote:
> > And globbing doesn't take care of all of it: a Linux program still has 
> > to iterate over a loop of filenames. The same as on Windows, except the 
> > latter will need to call a function to deliver the next filename.
> 
> Actually, most of them will require *two* loops, one to
> iterate over a sequence of filespecs, and another for
> the files matching each filespec.

Speaking of which, I think I've advocated before for fileinput to
perform these loops, and for some mechanism for argparse to do it,
etc... seems like it's about that time again.

There are other issues, like needing a way to do Windows' version of
wildcard parsing with all its quirks, or at least some of its quirks -
"*.*" for all files and "*." for files not containing any dot being the
most commonly used in the real world.

> Whereas the unix program only requires one loop.


From random832 at fastmail.com  Thu Dec  8 10:54:24 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 08 Dec 2016 10:54:24 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <1481212464.866079.812736153.341592F1@webmail.messagingengine.com>

On Wed, Dec 7, 2016, at 00:15, Steven D'Aprano wrote:
> and the shell expands the metacharacters ? {...} * [...] regardless of
> how much 
> smarts the command itself has.
> 
> There are thousands of programs I might use, and they may implement who
> knows 
> how many different globbing rules:
> 
> - some might support * and ? but not {...}

Just to point out, brace expansion isn't globbing. The most important
difference is that brace expansion doesn't care what files exist. {a,b}c
becomes "ac bc" regardless of if the files exist. {a,b}* becomes a* b*,
and if no files match one or both of these, it does whatever the shell
does in such cases.

This is why you can do commands like "mv
fix_a_ty{op,po}_in_this_filename"


From random832 at fastmail.com  Thu Dec  8 10:57:31 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 08 Dec 2016 10:57:31 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o28ifv$ros$1@blaine.gmane.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
Message-ID: <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>

On Wed, Dec 7, 2016, at 03:50, Peter Otten wrote:
> Is there an equivalent to
> 
> # touch -- -r 
> 
> on Windows?

Doesn't need one - options conventionally start with /, and filenames
can't contain /.


From cantorp at gmail.com  Thu Dec  8 11:30:14 2016
From: cantorp at gmail.com (cantorp at gmail.com)
Date: Thu, 8 Dec 2016 08:30:14 -0800 (PST)
Subject: Get min and max dates
In-Reply-To: <o2bo6j$ths$2@dont-email.me>
References: <o29cgh$nlj$3@dont-email.me>
 <5848ecc5$0$1605$c3e8da3$5496439d@news.astraweb.com>
 <o2bo6j$ths$2@dont-email.me>
Message-ID: <5ce7e20e-3041-49bf-8b63-62d1e289f09b@googlegroups.com>

Am Donnerstag, 8. Dezember 2016 14:47:31 UTC+1 schrieb DFS:
> On 12/8/2016 12:16 AM, Steven D'Aprano wrote:
> > On Thursday 08 December 2016 03:15, DFS wrote:
> >
> >> dts=  ['10-Mar-1998',
> >>         '20-Aug-1997',
> >>         '06-Sep-2009',
> >>         '23-Jan-2010',
> >>         '12-Feb-2010',
> >>         '05-Nov-2010',
> >>         '03-Sep-2009',
> >>         '07-Nov-2014',
> >>         '08-Mar-2013']
> >>
> >> Of course, the naive:
> >> min(dates) = '03-Sep-2009'
> >> max(dates) = '23-Jan-2010'
> >> is wrong.
> >>
> >> Not wanting to use any date parsing libraries, I came up with:
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > That's where you have gone wrong. By not using a date parsing library which
> > works and has been tested, you have to write your own dodgy and possibly buggy
> > parsing routine yourself.
> >
> > Why reinvent the wheel?
> 
> 
> Because the "wheel" is a pain in the ass.
> 

Why? And why do you use this wording?

> --------------------------------------------------------------
> import time
> dts=['10-Mar-1908','20-Aug-1937','06-Sep-1969','23-Jan-1952']
> def date_to_seconds(string):
>      return time.mktime(time.strptime(string, '%d-%b-%Y'))
> print min(dts, key=date_to_seconds)
> --------------------------------------------------------------
> 
> OverflowError: mktime argument out of range
> 
> 

(snip)

> 
> I like that flexibility, but mktime is apparently useless for dates 
> prior to 'the epoch'.

With a little more experience in Python programming you should have discovered that time.mktime is not even required to do your calculations.

Please remove time.mktime from the function date_to_seconds and rename the function to date_to_timestruct.

-- Paolo


From grant.b.edwards at gmail.com  Thu Dec  8 11:34:27 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Thu, 8 Dec 2016 16:34:27 +0000 (UTC)
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
Message-ID: <o2c22j$pin$1@blaine.gmane.org>

On 2016-12-08, Random832 <random832 at fastmail.com> wrote:
> On Wed, Dec 7, 2016, at 03:50, Peter Otten wrote:
>> Is there an equivalent to
>> 
>> # touch -- -r 
>> 
>> on Windows?
>
> Doesn't need one - options conventionally start with /, and filenames
> can't contain /.

But _paths_ can, and Windows command-line apps and shells choke on
paths when written with "/" separators because way-back when the
MS-DOS "designers" decided to use "/" as the default option character.

So, to avoid _that_ problem, Windows command line apps and the cmd.exe
shell only allow "\" as a path separator.

Back in the day, you could change the DOS option character to "-", and
then type paths they way God intended: with "/" as the separator.

-- 
Grant Edwards               grant.b.edwards        Yow! I want another
                                  at               RE-WRITE on my CEASAR
                              gmail.com            SALAD!!



From torriem at gmail.com  Thu Dec  8 11:35:42 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 09:35:42 -0700
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on Linux
 behaving differently than on Windows
In-Reply-To: <Jzc2A.281652$kM7.92263@fx44.am4>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
Message-ID: <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>

On 12/08/2016 05:43 AM, BartC wrote:
> (Neither will see Shift+Ctrl+B, which means go to start of the file, 
> same as Ctrl+Home. Ubuntu sees Ctrl+Home, but not Debian, although it 
> will report Alt+Home. And some laptop keyboards already have Home on an 
> Alternate-Function shift! It's a mess.)

Yes terminal-emulators on Linux are very different than consoles on
Windows because of a ton of historical baggage.  Unlike a Windows
console, ttys on Unix are designed to connect to serial lines that are
essentially 2-way data pipes. Thus all keyboard handling has to be
wrapped up in escape codes that travel along with the screen data down
those pipes.

Now you may consider this an obsolete model.  But the abstraction is
still particularly useful in the linux world as I can connect my
terminal emulator to a variety of mechanisms including remote ssh
connections, plain old telnet, actual serial lines, etc.  You may not
find that useful, but like I've said in this thread, I use this
functionality every day.  In Windows I would instead use a dedicated
program like Putty to access these various communication streams and
interpret them.

The reason you are seeing such different codes is because there are
different schemes for terminals.  Back in the day we dealt with real
terminals.  A little CRT with a keyboard and a serial link.  The
terminal would interpret various codes coming in the serial link and do
things like move the cursor, underline, bold, inverse, etc.  Different
companies developed their own competing terminals that had some feature
they thought people would like.  So we had vt100, vt200, Sun, Dec, ANSI,
and other terminal types. Some terminals could even interpret escape
codes and draw vector graphics on the CRT.  Unix had to support them all
because people used all kinds of different terminals even in the same
environment.

Now we mostly just use one terminal type, "xterm."  But you can still
find variations as you've found. But if you have the correct TERM
environment variable set (it usually is), you can and should use a
terminal library to decode and encode your escape sequences.

> 
>>> Except that was only two Linuxes; perhaps on others, the keyboard will
>>> likely be crippled in some other way.
>>
>> No, they'll all be the same -- if it has an ASCII code,
>> you'll be able to get it from a tty device, otherwise you
>> won't.
>>
> 
>>> How people manage to do anything on such an OS I've no idea.

The best way to handle terminals is to use a terminal library that
understands many different terminal types (encodings).  For example,
instead of getting a raw keyboard ascii code that could vary from
terminal emulation to terminal emulation, use ncurses to process the
keystrokes and give you some kind of normalized sequence.

Alternatively pick one terminal type and require all your users to use
that type of terminal.

But just know that all terminal types share some limitations with
regards to the modifier keys.  Some you just can't get via the terminal
interface.

> How do they work; what magic do they use to get that key information, 
> and why can't it be done outside of a GUI? As I understand a Linux GUI 
> is built on top of Linux.

X windows came much later than terminals and they built a more robust
protocol that allows client programs to access full keyboard state
information.

Applications running on a terminal don't have this information solely
because there's never been a facility for encoding this information into
escape codes.  There's no other great technical reason for this
limitation.  It's entirely possible that we could create a new,
non-standard terminal type that supported the shift modifier.

> # Python 2 because of the 'print' handling
> 
> def getch():     # adapted from first getch I saw on the internet
>          import sys, tty, termios
>          fd = sys.stdin.fileno()
>          old_settings = termios.tcgetattr(fd)
>          tty.setraw(sys.stdin.fileno())
>          ch = sys.stdin.read(1)
>          termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
>          return ord(ch)
> 
> print "Press keys"
> print ("Hit Escape twice to quit")
> 
> escape=0
> while 1:
>      k=getch()
> 
>      if k==27:
>          print
>          print "Esc ",
>          if escape: break
>      elif k==13:
>          print "Enter"
>      elif k==10:
>          print "Newline"
>      elif k<=26:
>          print "Ctrl",chr(k+64)
>      else:
>          print chr(k),
> 
>      escape = k==27
> 
> ----------------------

Raw terminal handling is indeed crufty!  That's why I prefer to use
ncurses when I need anything more basic than standard in and standard out.

> (On another test, using a C-based getch(), pressing Enter returns code 
> 10 not 13; another difference to note. Either code appears to clash with 
> Ctrl-M or Ctrl-J, a difference from Windows where Ctrl-J, Ctrl-M and 
> Enter are distinct keys, as they are in actuality.)

Interesting.  I wouldn't have thought ENTER would return a line feed.
What is your TERM environment variable set to?  If you switch to a
text-mode virtual console, you'd probably find yet more variations.

Yes Control codes are, well control codes.  Any ascii value under 32.
They are more or less common across terminal types.  I don't know of any
way around that with terminals.

In any case, don't worry about raw values. Use a terminal handling
library to give you something sane to work with.


From p.f.moore at gmail.com  Thu Dec  8 11:41:50 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 8 Dec 2016 08:41:50 -0800 (PST)
Subject: calling a program from Python batch file
In-Reply-To: <mailman.139.1481134988.2253.python-list@python.org>
References: <f9d6e83d-d8e7-4245-a02f-9065572632a9@googlegroups.com>
 <o29imb$h3i$1@reader1.panix.com>
 <033590e2-2717-76e2-73a0-462df35280ec@gmail.com>
 <mailman.139.1481134988.2253.python-list@python.org>
Message-ID: <53435410-cbc3-40d1-aafb-de7d38354130@googlegroups.com>

On Wednesday, 7 December 2016 18:23:23 UTC, Michael Torrie  wrote:

> Does Pexpect work on Windows?

Apparently yes: https://pexpect.readthedocs.io/en/stable/overview.html#pexpect-on-windows

"New in version 4.0: Windows support"

> In the OP's case it looks like the standard in pipe is sufficient.

Agreed.
Paul


From random832 at fastmail.com  Thu Dec  8 11:48:32 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 08 Dec 2016 11:48:32 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o29rde$ft9$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <6RZ1A.155334$EX4.96179@fx22.am4>
 <o29rde$ft9$1@dont-email.me>
Message-ID: <1481215712.878019.812787089.63BC5C8D@webmail.messagingengine.com>

On Wed, Dec 7, 2016, at 15:29, Lew Pitcher wrote:
> But, point of fact is that the feature to disable globbing is not often
> needed. Most Unix programs that accept filenames are happy to accept a
> list of filenames. There is not much call for a program to perform it's own
> globbing, like is required in Windows.
> 
> In fact, I can only think of three Unix "commandline" programs that need
> shell globbing disabled:
>   find - which performs it's own filename matching
>   grep - which uses regular expressions to search the contents of files,
>   and
>   sed  - which uses regular expressions to edit the contents of files.
> (I'm sure that there are a few more examples, though).

tar can do its own filename matching in some contexts, to match files
inside the archive for example.

7z does its own filename matching to allow distinguishing "extract from
multiple archives" [x \*.zip] from "extract a list of filenames from a
single archive" [x a.zip b.zip] - a BartC-compliant command line
paradigm if I ever saw one. I suspect it also allows you to match files
inside the archives in the list part.

scp lets you pass glob patterns to be matched on the remote server.
also, quoting for scp is somewhat unpleasant, since metacharacters in
general, not just globs but also $variables, quotes,`commands` etc, are
interpreted by both the local shell and the remote shell. sftp is a
little more sane, but still has remote globs for fetching, and quotes to
escape those globs.


From lew.pitcher at digitalfreehold.ca  Thu Dec  8 11:55:36 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Thu, 08 Dec 2016 11:55:36 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <6RZ1A.155334$EX4.96179@fx22.am4>
 <o29rde$ft9$1@dont-email.me>
 <1481215712.878019.812787089.63BC5C8D@webmail.messagingengine.com>
 <mailman.161.1481215715.2253.python-list@python.org>
Message-ID: <o2c37n$5c4$1@dont-email.me>

On Thursday December 8 2016 11:48, in comp.lang.python, "Random832"
<random832 at fastmail.com> wrote:

> On Wed, Dec 7, 2016, at 15:29, Lew Pitcher wrote:
>> But, point of fact is that the feature to disable globbing is not often
>> needed. Most Unix programs that accept filenames are happy to accept a
>> list of filenames. There is not much call for a program to perform it's own
>> globbing, like is required in Windows.
>> 
>> In fact, I can only think of three Unix "commandline" programs that need
>> shell globbing disabled:
>>   find - which performs it's own filename matching
>>   grep - which uses regular expressions to search the contents of files,
>>   and
>>   sed  - which uses regular expressions to edit the contents of files.
>> (I'm sure that there are a few more examples, though).
> 
> tar can do its own filename matching in some contexts, to match files
> inside the archive for example.
> 
> 7z does its own filename matching to allow distinguishing "extract from
> multiple archives" [x \*.zip] from "extract a list of filenames from a
> single archive" [x a.zip b.zip] - a BartC-compliant command line
> paradigm if I ever saw one. I suspect it also allows you to match files
> inside the archives in the list part.
> 
> scp lets you pass glob patterns to be matched on the remote server.
> also, quoting for scp is somewhat unpleasant, since metacharacters in
> general, not just globs but also $variables, quotes,`commands` etc, are
> interpreted by both the local shell and the remote shell. sftp is a
> little more sane, but still has remote globs for fetching, and quotes to
> escape those globs.


True. I had forgotten those.

Still, it's a short list of programs that either need to do their own
globbing, or need the shell to NOT glob for them. 

-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From random832 at fastmail.com  Thu Dec  8 11:55:58 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 08 Dec 2016 11:55:58 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
Message-ID: <1481216158.881072.812809001.65857A22@webmail.messagingengine.com>

On Wed, Dec 7, 2016, at 22:41, Steven D'Aprano wrote:
> Python's fnmatch lib is a good example. It has, or at least had, no
> support for escaping metacharacters. Anyone relying on Python's fnmatch and glob
> modules alone for globbing will be unable to handle legitimate file names.

That's not true. You can "escape" metacharacters by enclosing them in
square brackets, forming a character class of a single character. I've
done the same in SQL, so it occurred to me immediately. But of course
you have to know you have to do this, and it's not immediately obvious
that it will work for the square bracket characters themselves. Other
implementations of globbing (including native windows) don't do this and
don't even *have* square brackets as metacharacters.


From eryksun at gmail.com  Thu Dec  8 12:02:47 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 8 Dec 2016 17:02:47 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2c22j$pin$1@blaine.gmane.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
Message-ID: <CACL+1at54rOEurzRuSw8ZODC_iWP=vuSDFU=tqvyTKJH71kP0A@mail.gmail.com>

On Thu, Dec 8, 2016 at 4:34 PM, Grant Edwards <grant.b.edwards at gmail.com> wrote:
>
> So, to avoid _that_ problem, Windows command line apps and the cmd.exe
> shell only allow "\" as a path separator.

In cmd you can usually clarify the intent with quotes, e.g. `dir
C:/Windows` fails because it parses "Windows" as a parameter, but `dir
"C:/Windows"` succeeds.


From grant.b.edwards at gmail.com  Thu Dec  8 12:05:22 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Thu, 8 Dec 2016 17:05:22 +0000 (UTC)
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
Message-ID: <o2c3si$7lg$1@blaine.gmane.org>

On 2016-12-08, Michael Torrie <torriem at gmail.com> wrote:

> Now we mostly just use one terminal type, "xterm."

Or various other terminal emulators tha are mostly ANSI and Unicode
aware...

And the Linux console...

It's interesting to note that the "real" xterm terminal emulator will
still emulate a Tektronix storage-scope graphics terminal (a terminal
that actually _implemented_ vector drawing rather than emulating it
with a raster-scanned array of pixels).

But, I know of plenty of people that still use real serial terminals
connected via serial ports.  Until very recently, one of my customers
had a group of techs that bought used Wyse 50 (green monochrome CRT)
serial terminals off e-bay, Craigsist, or wherever and refurbished
them in-house so that they could be sent out to the field for
installation (as retail POS terminals).  Last time I heard, they were
goint to switch to flat-screen "thin-clients".  I got a chance to play
with one of those, and it was an industrial Single-board-PC and LCD
monitor built into a single unit running Windows.  It was configured
to boot up and run a Wyse-50 terminal emulator -- and connect to the
"real" computer via an RS-232C serial port. :)

-- 
Grant Edwards               grant.b.edwards        Yow! There's enough money
                                  at               here to buy 5000 cans of
                              gmail.com            Noodle-Roni!



From torriem at gmail.com  Thu Dec  8 12:08:43 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 10:08:43 -0700
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux behaving differently than on Windows
In-Reply-To: <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
Message-ID: <423687de-4ce8-4e48-fb59-eff127b132d6@gmail.com>

On 12/08/2016 09:35 AM, Michael Torrie wrote:
> Yes Control codes are, well control codes.  Any ascii value under 32.
> They are more or less common across terminal types.  I don't know of any
> way around that with terminals.

That is to say that on all terminal types that I'm aware of, the ENTER
key is encoded as Control-M, or 13.  Since they are the same key,
there's no way in any terminal to differentiate between pressing
control-M and the enter key.  I guess this limitation has been with us
so long that people did other things.  In unix terminal apps, often the
meta key is used as a modifier.  This is usually the "Alt" key.

If you need to differentiate between ENTER and control-M specifically,
you'll have to build a GUI app instead.  One that won't be able to run
remotely over a tty stream.

I know you want your apps to be exactly the same across platforms, but
that's not always possible, nor always desirable. For example, on Macs,
control-key is not normally used, but rather the Command-key (the apple
key) which happens to be where the Alt key is on our PC keyboards.




From grant.b.edwards at gmail.com  Thu Dec  8 12:16:29 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Thu, 8 Dec 2016 17:16:29 +0000 (UTC)
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <CACL+1at54rOEurzRuSw8ZODC_iWP=vuSDFU=tqvyTKJH71kP0A@mail.gmail.com>
Message-ID: <o2c4hd$f9f$1@blaine.gmane.org>

On 2016-12-08, eryk sun <eryksun at gmail.com> wrote:
> On Thu, Dec 8, 2016 at 4:34 PM, Grant Edwards <grant.b.edwards at gmail.com> wrote:
>>
>> So, to avoid _that_ problem, Windows command line apps and the cmd.exe
>> shell only allow "\" as a path separator.
>
> In cmd you can usually clarify the intent with quotes, e.g. `dir
> C:/Windows` fails because it parses "Windows" as a parameter, but `dir
> "C:/Windows"` succeeds.

That works to get the "/" past cmd.exe's parser and with seems to work
with cmd.exe's built-in commands, but programs that aren't internal to
cmd.exe become an issue.

-- 
Grant Edwards               grant.b.edwards        Yow! Here I am in 53
                                  at               B.C. and all I want is a
                              gmail.com            dill pickle!!



From torriem at gmail.com  Thu Dec  8 12:22:58 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 10:22:58 -0700
Subject: PonyORM: generators as a query syntax
In-Reply-To: <CA+M9+8US6PH24LrF_PhzZfX1Zx4QnOrXQC8t_7fNgPbJxOCxEQ@mail.gmail.com>
References: <2136a29c-d356-81e2-16e4-7d0b3745afdf@gmail.com>
 <CA+M9+8US6PH24LrF_PhzZfX1Zx4QnOrXQC8t_7fNgPbJxOCxEQ@mail.gmail.com>
Message-ID: <c547e418-afe9-c3a7-d890-a45ea7d88d12@gmail.com>

On 12/08/2016 07:26 AM, Alex Kaye wrote:
> Can you describe some uses or example for using ORM for a Newbie ?

Simply put, ORM is a method for making objects that represent records in
a database.  It's usually done in such a way that the objects are
"live."  In other words if the object has an attribute representing a
column in a table, if I assign to that, it will update the database
automatically. As well ORM abstracts and encapsulates relations between
database tables.  So if two tables are related, if you delete an object
from the first table, it will automatically ask the database to delete
dependent, related records from the related table.

https://en.wikipedia.org/wiki/Object-relational_mapping

Some people think ORM is pointless and would rather just work directly
with the databases using purpose query languages like SQL.  One reason
Django and other web frameworks use ORM is that it abstracts the
specifics of the database engine so you can more easily switch from
sqlite to mysql to posgresql or even use a commercial database engine.
There has always been a lively debate over ORM.  At some point the
abstraction will leak and you'll get some database-specific problem you
have to deal with in your app code.

For PonyORM examples, see the pony orm web pages:
https://docs.ponyorm.com/firststeps.html
https://docs.ponyorm.com/queries.html

Currently I'm planning to use it in a non-web application.  I messed
with it a bit last night and it works pretty well for what I need to use
it for, which is pretty simple on the database end.




From rosuav at gmail.com  Thu Dec  8 12:24:37 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 04:24:37 +1100
Subject: Name resolution and the (wrong?) LEGB rule
In-Reply-To: <58497384.9020900@oa-cagliari.inaf.it>
References: <58497384.9020900@oa-cagliari.inaf.it>
Message-ID: <CAPTjJmqmrGODQtJeYwJEgcMpBLopUwRtiYoQaybfJZt3Ju+9rA@mail.gmail.com>

On Fri, Dec 9, 2016 at 1:51 AM, Marco Buttu <marco.buttu at gmail.com> wrote:
> "if a particular name:object mapping cannot be found in the local
> namespaces, the namespaces of the enclosed scope are being searched next. If
> the search in the enclosed scope is unsuccessful, too, Python moves on to
> the global namespace, and eventually, it will search the built-in namespace
> (side note: if a name cannot found in any of the namespaces, a NameError
> will is raised)."
>
> AFAIK, Python has static scoping: the scope for a name is given during the
> bytecode compilation. This means that before executing the program, Python
> already know in which namespace to look for an object. So, it seems to me
> that the LEGB rule is wrong,

It isn't wrong, but there are some parts of it that can be resolved at
compile time. Once a function is compiled, it cannot normally gain or
lose local names. There are a few situations that can mess with that
(a star import or 'exec' statement, in Python 2), and when the
compiler detects one of those, it has to avoid the usual optimization.

The "usual optimization" is exactly what you describe: that different
bytecodes represent Local, Enclosing, and Global/Built-in scope
lookups. (Globals can be created or removed at run-time, so there's no
optimization possible there.) But in terms of language specifications,
the lookup rules are the same; it's just that the CPython compiler
takes advantage of information that it can see ("these are the only
locals for this function") to speed up execution.

ChrisA


From torriem at gmail.com  Thu Dec  8 12:27:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 10:27:39 -0700
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux behaving differently than on Windows
In-Reply-To: <o2c3si$7lg$1@blaine.gmane.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
 <o2c3si$7lg$1@blaine.gmane.org>
Message-ID: <87c9f0d6-aed9-1c0d-62df-19e4ab944f80@gmail.com>

On 12/08/2016 10:05 AM, Grant Edwards wrote:
> Or various other terminal emulators tha are mostly ANSI and Unicode
> aware...
> 
> And the Linux console...

True.

> 
> It's interesting to note that the "real" xterm terminal emulator will
> still emulate a Tektronix storage-scope graphics terminal (a terminal
> that actually _implemented_ vector drawing rather than emulating it
> with a raster-scanned array of pixels).

Yeah I played around with that for fun a few months ago.  The reason I
did was that someone on a forum was complaining about a problem with
that mode and I had never heard of it before!

> But, I know of plenty of people that still use real serial terminals
> connected via serial ports.  Until very recently, one of my customers
> had a group of techs that bought used Wyse 50 (green monochrome CRT)
> serial terminals off e-bay, Craigsist, or wherever and refurbished
> them in-house so that they could be sent out to the field for
> installation (as retail POS terminals).  Last time I heard, they were
> goint to switch to flat-screen "thin-clients".  I got a chance to play
> with one of those, and it was an industrial Single-board-PC and LCD
> monitor built into a single unit running Windows.  It was configured
> to boot up and run a Wyse-50 terminal emulator -- and connect to the
> "real" computer via an RS-232C serial port. :)

Very interesting.




From python.list at tim.thechases.com  Thu Dec  8 12:30:48 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Thu, 8 Dec 2016 11:30:48 -0600
Subject: Detect Linux Runlevel
In-Reply-To: <877f7cpaqr.fsf@elektro.pacujo.net>
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <20161206190043.7fc99068@bigbox.christie.dr>
 <mailman.105.1481072866.2253.python-list@python.org>
 <877f7cpaqr.fsf@elektro.pacujo.net>
Message-ID: <20161208113048.02875749@bigbox.christie.dr>

On 2016-12-07 07:30, Marko Rauhamaa wrote:
> Tim Chase <python.list at tim.thechases.com>:
> > On 2016-12-07 00:29, Marko Rauhamaa wrote:
> >> A word a warning: your code doesn't lock /var/run/utmp before
> >> access, which is a race condition. The file may be updated at any
> >> time, and ordinary file reads may yield corrupted records.
> >
> > Since the code is reading in record-sized blocks and never
> > writing, I'm not sure how much possibility there is for a race
> > condition. At worst, I imagine that it would result in reading
> > the old data which isn't a horrible condition.
> 
> If you read a full record at an offset, you might be right. However,
> your code uses Python's buffered I/O:
> 
>       with open(utmp_fname, "rb") as f:
>           while True:
>               bytes = f.read(XTMP_STRUCT_SIZE)
> 
> instead of os.open() and os.read().

Interesting.  I read up on os.open() and os.read() 
https://docs.python.org/2/library/os.html#os.read
but didn't notice anything there clarifying that it was unbuffered
compared to the __builtins__.open() and fp.read() functions.

Could you point me to resources where I can learn more about the
distinctions?

> > For under a certain block-size (PIPE_BUF, which I think used to be
> > the minimum POSIX requirement of 512b, but is now 4096b on Linux),
> > *nix operating systems were atomic in their reading and writing.
> 
> That particular guarantee is true for pipes and FIFOs only.

Ah, my error in misreading that as globally applicable.  Thanks for
ensuring accuracy.

-tkc




From skip.montanaro at gmail.com  Thu Dec  8 13:01:20 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 8 Dec 2016 12:01:20 -0600
Subject: Get min and max dates
In-Reply-To: <5ce7e20e-3041-49bf-8b63-62d1e289f09b@googlegroups.com>
References: <o29cgh$nlj$3@dont-email.me>
 <5848ecc5$0$1605$c3e8da3$5496439d@news.astraweb.com>
 <o2bo6j$ths$2@dont-email.me>
 <5ce7e20e-3041-49bf-8b63-62d1e289f09b@googlegroups.com>
Message-ID: <CANc-5UwCyEKKcYmh2Y6PTY1tzMB0jwWYUtJGafKZ1v2+JGd6+w@mail.gmail.com>

Datetime has greater range I believe. I can't paste from my work computer,
but try this:

min(datetime.datetime.strptime(s, "%d-%b-%Y") for s in dts)

You should get the 1908 date instead of the 1969 date.

In general, you should always use datetime instead of time for doing date
manipulation and date arithmetic. It's a *whole lot better*.

Skip


From __peter__ at web.de  Thu Dec  8 13:03:38 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 08 Dec 2016 19:03:38 +0100
Subject: Name resolution and the (wrong?) LEGB rule
References: <58497384.9020900@oa-cagliari.inaf.it>
 <CAPTjJmqmrGODQtJeYwJEgcMpBLopUwRtiYoQaybfJZt3Ju+9rA@mail.gmail.com>
Message-ID: <o2c79m$rpb$1@blaine.gmane.org>

Chris Angelico wrote:

> On Fri, Dec 9, 2016 at 1:51 AM, Marco Buttu <marco.buttu at gmail.com> wrote:
>> "if a particular name:object mapping cannot be found in the local
>> namespaces, the namespaces of the enclosed scope are being searched next.
>> If the search in the enclosed scope is unsuccessful, too, Python moves on
>> to the global namespace, and eventually, it will search the built-in
>> namespace (side note: if a name cannot found in any of the namespaces, a
>> NameError will is raised)."
>>
>> AFAIK, Python has static scoping: the scope for a name is given during
>> the bytecode compilation. This means that before executing the program,
>> Python already know in which namespace to look for an object. So, it
>> seems to me that the LEGB rule is wrong,

It might be sufficient to say that the LE part is usually statically 
determined while the GB part is dynamic. 

The odd beast is the class namespace:
 
>>> x = "outer"
>>> class C:
...     print(x)
...     x = "inner"
...     print(x)
...     del x
...     print(x)
... 
outer
inner
outer

> It isn't wrong, but there are some parts of it that can be resolved at
> compile time. Once a function is compiled, it cannot normally gain or
> lose local names. There are a few situations that can mess with that
> (a star import or 'exec' statement, in Python 2), and when the
> compiler detects one of those, it has to avoid the usual optimization.
> 
> The "usual optimization" is exactly what you describe: that different
> bytecodes represent Local, Enclosing, and Global/Built-in scope
> lookups. (Globals can be created or removed at run-time, so there's no
> optimization possible there.) But in terms of language specifications,
> the lookup rules are the same; it's just that the CPython compiler
> takes advantage of information that it can see ("these are the only
> locals for this function") to speed up execution.

If it is only an optimization why doesn't a failing local lookup fall back 
to the global namespace?



From marko at pacujo.net  Thu Dec  8 13:07:16 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 08 Dec 2016 20:07:16 +0200
Subject: Detect Linux Runlevel
References: <I6-dnWkd05mNSNjFnZ2dnUU7-e_NnZ2d@giganews.com>
 <20161205160857.70bff64d@bigbox.christie.dr>
 <mailman.58.1480976852.2253.python-list@python.org>
 <CtOdnQ4HvtpWmNvFnZ2dnUU7-QmdnZ2d@giganews.com>
 <20161205214252.594f722f@bigbox.christie.dr>
 <mailman.75.1480995784.2253.python-list@python.org>
 <87d1h4pu8j.fsf@elektro.pacujo.net>
 <20161206190043.7fc99068@bigbox.christie.dr>
 <mailman.105.1481072866.2253.python-list@python.org>
 <877f7cpaqr.fsf@elektro.pacujo.net>
 <20161208113048.02875749@bigbox.christie.dr>
 <mailman.170.1481218263.2253.python-list@python.org>
Message-ID: <87bmwm1ei3.fsf@elektro.pacujo.net>

Tim Chase <python.list at tim.thechases.com>:

> Interesting.  I read up on os.open() and os.read() 
> https://docs.python.org/2/library/os.html#os.read
> but didn't notice anything there clarifying that it was unbuffered
> compared to the __builtins__.open() and fp.read() functions.
>
> Could you point me to resources where I can learn more about the
> distinctions?

It is not explained very clearly in the documentation, but the os.*
functions are thin wrappers around the analogous C functions (system
calls or standard library functions).

There is just this allusion:

   Note that using the file descriptor directly will bypass the file
   object methods, ignoring aspects such as internal buffering of data.

   <URL: https://docs.python.org/3/library/os.html?#file-descriptor-op
   erations>

The os.* facilities are ultimately documented in the Linux man pages.


File object buffering can be turned off by adding buffering=0 to the
high-level open() builtin function:

   When no buffering argument is given, the default buffering policy
   works as follows:

    * Binary files are buffered in fixed-size chunks; the size of the
      buffer is chosen using a heuristic trying to determine the
      underlying device?s ?block size? and falling back on
      io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will typically
      be 4096 or 8192 bytes long.

   <URL: https://docs.python.org/3/library/functions.html#open>


In general, system programming is best done using system programming
facilities, ie, os.*.


Marko


From rosuav at gmail.com  Thu Dec  8 13:14:50 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 05:14:50 +1100
Subject: Name resolution and the (wrong?) LEGB rule
In-Reply-To: <o2c79m$rpb$1@blaine.gmane.org>
References: <58497384.9020900@oa-cagliari.inaf.it>
 <CAPTjJmqmrGODQtJeYwJEgcMpBLopUwRtiYoQaybfJZt3Ju+9rA@mail.gmail.com>
 <o2c79m$rpb$1@blaine.gmane.org>
Message-ID: <CAPTjJmpQO99bH3ozGQcimKH5emCKwUt6cSYUJXXod6CpBVmo-g@mail.gmail.com>

On Fri, Dec 9, 2016 at 5:03 AM, Peter Otten <__peter__ at web.de> wrote:
>> The "usual optimization" is exactly what you describe: that different
>> bytecodes represent Local, Enclosing, and Global/Built-in scope
>> lookups. (Globals can be created or removed at run-time, so there's no
>> optimization possible there.) But in terms of language specifications,
>> the lookup rules are the same; it's just that the CPython compiler
>> takes advantage of information that it can see ("these are the only
>> locals for this function") to speed up execution.
>
> If it is only an optimization why doesn't a failing local lookup fall back
> to the global namespace?

Define "failing". Do you mean that this should print "outer"?

x = "outer"
def f():
    print(x)
    x = "inner"
f()

There are plenty of languages where this is true, but they work
because the defined scope of a variable is "from its declaration
down". Python doesn't work like that. Neither does JavaScript,
although it's a bit bizarre in a few ways. The lookup doesn't fail; it
finds a local variable that doesn't have a value. At least, I'm pretty
sure that's how it works. Is there a language specification stating
this?

ChrisA


From plzr at mac.com  Thu Dec  8 13:23:12 2016
From: plzr at mac.com (3dB)
Date: Thu, 8 Dec 2016 10:23:12 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <mailman.153.1481188555.2253.python-list@python.org>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
Message-ID: <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>

Anton, Thanks for the detailed response. I will look into virtualenv and others. Please excuse beginner's question, but is Xcode of any use in this type of situation?


From torriem at gmail.com  Thu Dec  8 13:34:52 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 11:34:52 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
Message-ID: <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>

On 12/08/2016 11:23 AM, 3dB wrote:
> Anton, Thanks for the detailed response. I will look into virtualenv
> and others. Please excuse beginner's question, but is Xcode of any
> use in this type of situation?

Not directly, but it does provide a C compiler, which some Python
modules require when you build and install them.  So having Xcode
installed could be a requirement for installing some python modules (but
not all, nor is it required to run python scripts).

Are you asking if there are some nice IDEs available for working with
Python?


From __peter__ at web.de  Thu Dec  8 14:14:09 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 08 Dec 2016 20:14:09 +0100
Subject: Name resolution and the (wrong?) LEGB rule
References: <58497384.9020900@oa-cagliari.inaf.it>
 <CAPTjJmqmrGODQtJeYwJEgcMpBLopUwRtiYoQaybfJZt3Ju+9rA@mail.gmail.com>
 <o2c79m$rpb$1@blaine.gmane.org>
 <CAPTjJmpQO99bH3ozGQcimKH5emCKwUt6cSYUJXXod6CpBVmo-g@mail.gmail.com>
Message-ID: <o2cbdt$dh6$1@blaine.gmane.org>

Chris Angelico wrote:

> On Fri, Dec 9, 2016 at 5:03 AM, Peter Otten <__peter__ at web.de> wrote:
>>> The "usual optimization" is exactly what you describe: that different
>>> bytecodes represent Local, Enclosing, and Global/Built-in scope
>>> lookups. (Globals can be created or removed at run-time, so there's no
>>> optimization possible there.) But in terms of language specifications,
>>> the lookup rules are the same; it's just that the CPython compiler
>>> takes advantage of information that it can see ("these are the only
>>> locals for this function") to speed up execution.
>>
>> If it is only an optimization why doesn't a failing local lookup fall
>> back to the global namespace?
> 
> Define "failing". Do you mean that this should print "outer"?
> 
> x = "outer"
> def f():
>     print(x)
>     x = "inner"
> f()

I mean it could, as classes already work that way. I think the current 
behaviour is a design decision rather than an implementation accident.
 
> There are plenty of languages where this is true, but they work
> because the defined scope of a variable is "from its declaration
> down". Python doesn't work like that. Neither does JavaScript,
> although it's a bit bizarre in a few ways. The lookup doesn't fail; it
> finds a local variable that doesn't have a value. At least, I'm pretty
> sure that's how it works. Is there a language specification stating
> this?

I found

https://www.python.org/dev/peps/pep-0227/




From bc at freeuk.com  Thu Dec  8 14:15:01 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 19:15:01 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
Message-ID: <o2cbd8$5b2$1@dont-email.me>

On 08/12/2016 03:41, Steven D'Aprano wrote:
> On Thursday 08 December 2016 12:15, BartC wrote:
>

>> That's all. I know the value of keeping things straightforward instead
>> of throwing in everything you can think of. The file-matching is done by
>> WinAPI functions.

> So you're happy with the fact that there are legitimate file names that your
> program simply has no way of dealing with?

Perfectly. Such names are illegal on its home OS which is Windows, and 
ill-advised elsewhere.

Plus you're missing the fact that these are /functions/, not user 
interfaces, and are entitled to use whatever scheme they like to express 
patterns of filenames. Since it has few ambitions, it just uses 
Windows-style wildcards.

> Thank you for making my point for me! If you leave the implementation of
> metacharacter expansion

This is some feature most of my users had never heard of and never cared 
for. It only appears to exist in the CLI of some shell or other that 
runs on Linux.

> Python's fnmatch lib is a good example. It has, or at least had, no support for
> escaping metacharacters. Anyone relying on Python's fnmatch and glob modules
> alone for globbing will be unable to handle legitimate file names.

That was a good choice. This stuff might be OK for interactive user 
input in a shell. It might even be OK for sticking in a script file in 
the same language as that shell

But it DOESN'T BELONG in a general purpose library, written in a 
different language, and nothing to do with the language syntax used by 
that shell. It might *opt* to include some well-known elements (such as 
? and *), but that is a choice.

If you want full shell functionality within a program, then just invoke 
that shell!

>> Reading that post again, you presumably meant either *.* used to match
>> all files (with an embedded "." in Linux, with or without in Windows;
>> another difference), or *.* used to match a specific file called *.*.
>
> Correct.
>
> It doesn't matter whether metacharacters are expanded by the shell or the
> program itself, there needs to be an escaping mechanism for those cases where
> you want the metacharacters to be treated literally.

How do you write a specifier for filenames that consist of any 
characters but which must be in alphabetical order? (eg. 'forty' but not 
'fifty')

My point is it's easy to come up with a requirement that can't be 
handled. (If that one can, then I'll just think of a different one!)

(Here's one solution if I desperately needed that feature: 
http://pastebin.com/wjzu9L6L. Probably a Python one-liner given an input 
list of filenames.)

>>> So presumably your dirlist() command can distinguish between the file called
>>> literally "*.*" and the file spec "*.*" that should be expanded,
>>
>> No. I won't support that (not unless it's supported by Posix's
>> fnmatch()). Because it's the thin end of the wedge. I can show a few
>> lines of code and then you will say, Ah, but what about this...
>>
>> And I first used this function in early 90s I think, I don't recall it
>> ever not working.
>
> If you can't use it to specify a file called literally "*.*", then its not
> working.

Presumably your solution requires some intervention to turn *.* into 
something that is escaped so that it will be matched literally. But 
then, the same kind of intervention can be used to instead call 
dirlist_special():

function dirlist_special(pathandfile)=
     if checkfile(pathandfile) then
         return (extractfile(pathandfile),)
     else
         return ()
     fi
end

print dirlist_special("*.*")

>>> And of course your program is also capable of variable and arithmetic
>>> expansion, right?
>>
>> Um, dirlist() is used within a language, of course it can do all that.
>> If you mean within the file-pattern string submitted to dirlist, then I
>> don't even know what that would mean.
>
> I showed you an example in another post:
>
> [steve at ando ~]$ export base="thefile"
> [steve at ando ~]$ ls -l "$base$((1000 + 234))"
> -rw-rw-r-- 1 steve steve 0 Dec  8 10:51 thefile1234
>
>
> Of course it is silly to write 1000 + 234 as a literal constant like that, but
> either or both of those could just as easily be variables.
>
> The point here is not that *you* should build a calculator into your version of
> ls.

I don't have to do anything of the sort. I'm quite happy that the shell 
does that sort of esoteric stuff which wouldn't occur to many people to 
want to do (if they even know about it), and doesn't interfere with the 
basics.

Yes, I've written applications that use a command line, even GUI ones, 
which allows variables and expressions. But that is relevant in the 
context of that application and not outside .

Take all the programs that you run on Linux, /including GUI programs/. 
In all the situations where it requires a file-name and lets you type it 
in (not just from the shell command line, but within a program or from a 
GUI entry box), does it let you do all this meta-character and 
meta-expression stuff that you're on about?

If so, please tell me how to define '$base="thefile"' from inside 
LibreOffice, and how to make use of it in the file dialog entry box.

If not, then what you're arguing about is meaningless.

> For example, a quick and simple way of backing up a file with a date stamp:
>
> steve at runes:~$ cp alloc.txt{,-`date +%Y%m%d`}
> steve at runes:~$ ls alloc.txt*
> alloc.txt  alloc.txt-20161208

Pretty good. Except I just tried that method in Firefox to save a page:

c.html{,-`date +%Y%m%d`}

and it worked - sort of. I just got a saved page called:

c.html{,-`date +%Y%m%d`}

Does this mean Firefox is broken?


-- 
Bartc


From rosuav at gmail.com  Thu Dec  8 14:36:00 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 06:36:00 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2cbd8$5b2$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
Message-ID: <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>

On Fri, Dec 9, 2016 at 6:15 AM, BartC <bc at freeuk.com> wrote:
> On 08/12/2016 03:41, Steven D'Aprano wrote:
>>
>> On Thursday 08 December 2016 12:15, BartC wrote:
>>
>
>>> That's all. I know the value of keeping things straightforward instead
>>> of throwing in everything you can think of. The file-matching is done by
>>> WinAPI functions.
>
>
>> So you're happy with the fact that there are legitimate file names that
>> your
>> program simply has no way of dealing with?
>
>
> Perfectly. Such names are illegal on its home OS which is Windows, and
> ill-advised elsewhere.

Awesome. Since you're using Windows as your definition of
"ill-advised", I take it you assume that file names should be case
insensitive, too, right? Cool. Let me know when you've figured out the
uppercase and lowercase equivalents of this file name: "????iI????"

ChrisA


From HooDunnit at didly42KahZidly.net  Thu Dec  8 14:49:03 2016
From: HooDunnit at didly42KahZidly.net (Cousin Stanley)
Date: Thu, 08 Dec 2016 12:49:03 -0700
Subject: Get min and max dates
References: <o29cgh$nlj$3@dont-email.me>
Message-ID: <o2cdd0$ca3$1@dont-email.me>

DFS wrote:

> ....
> Not wanting to use any date parsing libraries,
> ....

  If you happen reconsider date parsing libraries 
  the  strptime  function  from the  datetime  module
  might be useful ....

#!/usr/bin/env python3

from datetime import datetime

dates = [ '10-Mar-1998' ,
          '20-Aug-1997' ,
          '06-Sep-2009' ,
          '23-Jan-2010' ,
          '12-Feb-2010' ,
          '05-Nov-2010' ,
          '03-Sep-2009' ,
          '07-Nov-2014' ,
          '08-Mar-2013' ]

dict_dates = { }

print( )

for this_date in dates :

    dt = datetime.strptime( this_date , '%d-%b-%Y' )

    sd = dt.strftime( '%Y-%m-%d' )

    print( '  {0} .... {1}'.format( this_date , sd ) )

    dict_dates[ sd ] = this_date


min_date = min( dict_dates.keys() )

max_date = max( dict_dates.keys() )

print( '\n  {0} .... {1}  min'.format( dict_dates[ min_date ] , min_date ) )

print( '\n  {0} .... {1}  max'.format( dict_dates[ max_date ] , max_date ) )








































-- 
Stanley C. Kitching
Human Being
Phoenix, Arizona



From plzr at mac.com  Thu Dec  8 15:26:20 2016
From: plzr at mac.com (3dB)
Date: Thu, 8 Dec 2016 12:26:20 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <mailman.174.1481222097.2253.python-list@python.org>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
Message-ID: <cfeb191c-806f-4fd9-8c9f-dcbbab46eb64@googlegroups.com>

Thanks Michael,

I think I was confusing Virtual Environment (VE) with IDE.

Probably best if I get to grips with virtualenv to start with since I can't complete installations at present and VE appears to offer best solution.


From plzr at mac.com  Thu Dec  8 15:48:35 2016
From: plzr at mac.com (3dB)
Date: Thu, 8 Dec 2016 12:48:35 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <mailman.174.1481222097.2253.python-list@python.org>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
Message-ID: <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>

Michael,

I tried installing virtual env but got similar permissions error:

IOError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/virtualenv.py'


From torriem at gmail.com  Thu Dec  8 16:10:06 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 14:10:06 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <cfeb191c-806f-4fd9-8c9f-dcbbab46eb64@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <cfeb191c-806f-4fd9-8c9f-dcbbab46eb64@googlegroups.com>
Message-ID: <1455ed2e-e88a-85a7-495b-7b247e354609@gmail.com>

On 12/08/2016 01:26 PM, 3dB wrote:
> Thanks Michael,
> 
> I think I was confusing Virtual Environment (VE) with IDE.
> 
> Probably best if I get to grips with virtualenv to start with since I
> can't complete installations at present and VE appears to offer best
> solution.

Despite Anton's warning about sudo, you definitely could use it to
install the module to /Library and get on with using it for now, and
learn about virtual environments once you've got some python experience.
 The problem was that the install function of the package wanted to
write files to that location in /Library, which non-root users cannot
do. Prefacing that command with sudo will elevate access temporarily
allowing the installation to proceed. That's what I would do if I were
you.

Who cares if you have to redo it after you reinstall the OS, or even
move to a different computer. Yes virtual environments are nice and
self-contained and require no special privileges to use.  But if you
don't know anything about python, that's going to be an advanced topic
for now.


From torriem at gmail.com  Thu Dec  8 16:11:11 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 14:11:11 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
Message-ID: <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>

On 12/08/2016 01:48 PM, 3dB wrote:
> Michael,
> 
> I tried installing virtual env but got similar permissions error:
> 
> IOError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/virtualenv.py'

Just to get virtual environment support you definitely have to install
this package, and it must go to the system packages area. So I think
sudo is a must for this one.

sudo <whatever command you typed before>




From greg.ewing at canterbury.ac.nz  Thu Dec  8 16:15:56 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 09 Dec 2016 10:15:56 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.157.1481212466.2253.python-list@python.org>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <1481212464.866079.812736153.341592F1@webmail.messagingengine.com>
 <mailman.157.1481212466.2253.python-list@python.org>
Message-ID: <eau0seFqts5U1@mid.individual.net>

Random832 wrote:
> Just to point out, brace expansion isn't globbing. The most important
> difference is that brace expansion doesn't care what files exist.

However, it's something that the shell expands into multiple
arguments, giving it similar characteristics for the purposes
of this discussion.

The main difference is that you're unlikely to accidentally
get a million file names or 131072 bytes of arguments that
way. :-)

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Dec  8 16:25:52 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 09 Dec 2016 10:25:52 +1300
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.159.1481214881.2253.python-list@python.org>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me> <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <mailman.159.1481214881.2253.python-list@python.org>
Message-ID: <eau1f2Fr2o8U1@mid.individual.net>

Grant Edwards wrote:
> But _paths_ can, and Windows command-line apps and shells choke on
> paths when written with "/" separators because way-back when the
> MS-DOS "designers" decided to use "/" as the default option character.

To be fair to them, the use of "/" for options can be traced
back to earlier systems such as CP/M and RT-11, which didn't
even have hierarchial file systems, so choice of path
separators wasn't an issue.

The "/" made a kind of sense in those systems, because options
were thought of as annotations attached to an argument rather
than as arguments in their own right. They were typically
written straight after the argument they applied to without
any space between, or after the command name itself if they
didn't apply to any particular argument.

-- 
Greg


From rosuav at gmail.com  Thu Dec  8 16:26:35 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 08:26:35 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <eau0seFqts5U1@mid.individual.net>
References: <o26pll$fpk$1@dont-email.me>
 <20161206214628.GA83577@cskk.homeip.net>
 <CAPTjJmqsKy5sW4P5qCwx-MhHj5M4DGGby_Dr7EGkG_x6kJ3bwg@mail.gmail.com>
 <CAPjBxg8A6JWPv00dDHjneiRwaGUC91JdK249vuz5LxEzX_aPXQ@mail.gmail.com>
 <mailman.99.1481063728.2253.python-list@python.org>
 <o27q4d$am8$1@dont-email.me>
 <58479af7$0$1526$c3e8da3$5496439d@news.astraweb.com>
 <1481212464.866079.812736153.341592F1@webmail.messagingengine.com>
 <mailman.157.1481212466.2253.python-list@python.org>
 <eau0seFqts5U1@mid.individual.net>
Message-ID: <CAPTjJmqpP6dGjT34hF15m3u-aa6wrVCMge4=pwD4i=jrCtjftA@mail.gmail.com>

On Fri, Dec 9, 2016 at 8:15 AM, Gregory Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> The main difference is that you're unlikely to accidentally
> get a million file names or 131072 bytes of arguments that
> way. :-)

python3 argcount.py
{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}{a,b}
65537

ChrisA


From skip.montanaro at gmail.com  Thu Dec  8 16:27:04 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 8 Dec 2016 15:27:04 -0600
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
Message-ID: <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>

Sorry, I haven't been following this thread carefully. Michael's use of
"sudo" caught my eye though. I know virtualenv might be "special", but
shouldn't this work?

pip install --user virtualenv

It will wind up in $HOME/.local/lib/python2.7/site-packages (or similar) I
believe.

Skip

On Thu, Dec 8, 2016 at 3:11 PM, Michael Torrie <torriem at gmail.com> wrote:

> On 12/08/2016 01:48 PM, 3dB wrote:
> > Michael,
> >
> > I tried installing virtual env but got similar permissions error:
> >
> > IOError: [Errno 13] Permission denied: '/Library/Python/2.7/site-
> packages/virtualenv.py'
>
> Just to get virtual environment support you definitely have to install
> this package, and it must go to the system packages area. So I think
> sudo is a must for this one.
>
> sudo <whatever command you typed before>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From torriem at gmail.com  Thu Dec  8 16:28:33 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 14:28:33 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
 <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
Message-ID: <a1a781fd-8bd6-a6e7-ca96-3e2a11c9eead@gmail.com>

On 12/08/2016 02:27 PM, Skip Montanaro wrote:
> Sorry, I haven't been following this thread carefully. Michael's use of
> "sudo" caught my eye though. I know virtualenv might be "special", but
> shouldn't this work?
> 
> pip install --user virtualenv
> 
> It will wind up in $HOME/.local/lib/python2.7/site-packages (or similar) I
> believe.

Ahh yes, that's probably correct.


From greg.ewing at canterbury.ac.nz  Thu Dec  8 16:38:29 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 09 Dec 2016 10:38:29 +1300
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux   behaving differently than on Windows
In-Reply-To: <mailman.160.1481214949.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
 <mailman.160.1481214949.2253.python-list@python.org>
Message-ID: <eau26nFr8bpU1@mid.individual.net>

Michael Torrie wrote:
> Interesting.  I wouldn't have thought ENTER would return a line feed.

Possibly you have the terminal in "cbreak" mode, which
provides a character at a time but still does things
like translate CR->LF.

-- 
Greg


From bc at freeuk.com  Thu Dec  8 16:42:25 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 21:42:25 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.176.1481225770.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
Message-ID: <o2ck1k$51c$1@dont-email.me>

On 08/12/2016 19:36, Chris Angelico wrote:
> On Fri, Dec 9, 2016 at 6:15 AM, BartC <bc at freeuk.com> wrote:
>> On 08/12/2016 03:41, Steven D'Aprano wrote:
>>>
>>> On Thursday 08 December 2016 12:15, BartC wrote:
>>>
>>
>>>> That's all. I know the value of keeping things straightforward instead
>>>> of throwing in everything you can think of. The file-matching is done by
>>>> WinAPI functions.
>>
>>
>>> So you're happy with the fact that there are legitimate file names that
>>> your
>>> program simply has no way of dealing with?
>>
>>
>> Perfectly. Such names are illegal on its home OS which is Windows, and
>> ill-advised elsewhere.
>
> Awesome. Since you're using Windows as your definition of
> "ill-advised", I take it you assume that file names should be case
> insensitive, too, right? Cool. Let me know when you've figured out the
> uppercase and lowercase equivalents of this file name: "????iI????"


Python3 tells me that original, lower-case and upper-case versions are:

????iI????
???i?ii????
SS?I?II????

(Python2 failed to run the code:

s="????iI????"
print (s)
print (s.lower())
print (s.upper())
)

But, given that, what's your point? That some esoteric Unicode 
characters have ill-defined upper and lower case versions, and therefore 
it is essential to treat them distinctly in EVERY SINGLE ALPHABET 
including English?

I guess that means that if I try a write a book about a character called 
HarrY potter or james BOND then I cannot be sued.

-- 
Bartc


From greg.ewing at canterbury.ac.nz  Thu Dec  8 16:46:00 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 09 Dec 2016 10:46:00 +1300
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux behaving differently than on Windows
In-Reply-To: <mailman.165.1481216929.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
 <423687de-4ce8-4e48-fb59-eff127b132d6@gmail.com>
 <mailman.165.1481216929.2253.python-list@python.org>
Message-ID: <eau2kqFrbefU1@mid.individual.net>

Michael Torrie wrote:
> For example, on Macs,
> control-key is not normally used, but rather the Command-key (the apple
> key) which happens to be where the Alt key is on our PC keyboards.

Actually, Alt is usually mapped to Option on a Mac. The Mac
Command key corresponds the "Windows" or "Meta" key on other
keyboards, I think (but is used in a very different way).

-- 
Greg


From rosuav at gmail.com  Thu Dec  8 17:31:55 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 09:31:55 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2ck1k$51c$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
Message-ID: <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>

On Fri, Dec 9, 2016 at 8:42 AM, BartC <bc at freeuk.com> wrote:
> Python3 tells me that original, lower-case and upper-case versions are:
>
> ????iI????
> ???i?ii????
> SS?I?II????

Now lower-case the upper-case version and see what you get. And
upper-case the lower-case version. Because x.upper().lower() should be
the same as x.lower(), right? And x.lower().upper().lower() is the
same too. Right?

>>> x = "????iI????"
>>> x.upper().lower() == x.lower()
False
>>> x.upper().lower() == x.lower().upper().lower()
False

> (Python2 failed to run the code:
>
> s="????iI????"
> print (s)
> print (s.lower())
> print (s.upper())
> )

I don't know what you mean by "failed", but you shouldn't have
non-ASCII characters in Python 2 source code without a coding cookie.
Also, you should be using a Unicode string. Or just stick with Py3.

> But, given that, what's your point? That some esoteric Unicode characters
> have ill-defined upper and lower case versions, and therefore it is
> essential to treat them distinctly in EVERY SINGLE ALPHABET including
> English?

Yes, because it's Unicode's fault, isn't it. The world was so nice and
simple before Unicode came along and created all these messes for us
to have to deal with. And you're quite right - these characters are
esoteric and hardly ever used. [1] And they're so ill-defined that
nobody could predict or figure out what the upper-case and lower-case
forms are. It's not like there are rules that come from the languages
where they're used.

> I guess that means that if I try a write a book about a character called
> HarrY potter or james BOND then I cannot be sued.

This is not legal advice, but I think you'll agree that "HarrY" is not
equal to "Harry". Whether it's a good idea to have two files in the
same directory that have those names, it's certainly the case that the
names can be distinguished. (Sir HarrY is a very distinguished name.)
And if you decide not to distinguish between "Y" and "y", then which
of the above characters should be not-distinguished?

ChrisA

[1] Okay, to be fair, one of the ones I used *is* esoteric. But most
of them aren't.


From marko at pacujo.net  Thu Dec  8 18:03:48 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 01:03:48 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <mailman.159.1481214881.2253.python-list@python.org>
 <eau1f2Fr2o8U1@mid.individual.net>
Message-ID: <87lgvqnhuz.fsf@elektro.pacujo.net>

Gregory Ewing <greg.ewing at canterbury.ac.nz>:

> Grant Edwards wrote:
>> But _paths_ can, and Windows command-line apps and shells choke on
>> paths when written with "/" separators because way-back when the
>> MS-DOS "designers" decided to use "/" as the default option
>> character.
>
> To be fair to them, the use of "/" for options can be traced back to
> earlier systems such as CP/M and RT-11, which didn't even have
> hierarchial file systems, so choice of path separators wasn't an
> issue.

I find it a bit annoying that "/" cannot be used in Linux filenames.
Slashes have very common everyday uses and it's a pity they are
reserved. Instead, ASCII control characters are allowed, which is of no
use.

And the Linux example of the other day is just terrible. Globs can do
surprising damage if filenames should start with a minus sign.

The naive command:

    cp *.c backup/

should really be:

    cp ./*.c backup/

or:

    cp -- *.c backup/

but who does that?


Marko


From bc at freeuk.com  Thu Dec  8 18:19:55 2016
From: bc at freeuk.com (BartC)
Date: Thu, 8 Dec 2016 23:19:55 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.182.1481236319.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
Message-ID: <o2cpof$r89$1@dont-email.me>

On 08/12/2016 22:31, Chris Angelico wrote:
> On Fri, Dec 9, 2016 at 8:42 AM, BartC <bc at freeuk.com> wrote:
>> Python3 tells me that original, lower-case and upper-case versions are:
>>
>> ????iI????
>> ???i?ii????
>> SS?I?II????
>
> Now lower-case the upper-case version and see what you get. And
> upper-case the lower-case version. Because x.upper().lower() should be
> the same as x.lower(), right? And x.lower().upper().lower() is the
> same too. Right?

I get this (although I suspect Thunderbird will screw up the tabs); the 
code I used follows at the end:

	 L	 U	L->U	U->L

A 	 a 	 A 	 A 	 a      Letters
65 	 97 	 65 	 65 	 97     Ordinals
1 	 1 	 1 	 1 	 1      Lengths
   	   	   	   	
32 	 32 	 32 	 32 	 32
1 	 1 	 1 	 1 	 1

? 	 ? 	 SS 	 SS 	 ss
223 	 223 	 83 	 83 	 115
1 	 1 	 2 	 2 	 2

? 	 ? 	 ? 	 SS 	 ?
7838 	 223 	 7838 	 83 	 223
1 	 1 	 1 	 2 	 1

? 	 ? 	 I 	 I 	 i
305 	 305 	 73 	 73 	 105
1 	 1 	 1 	 1 	 1

? 	 i? 	 ? 	 I? 	 i?
304 	 105 	 304 	 73 	 105
1 	 2 	 1 	 2 	 2

i 	 i 	 I 	 I 	 i
105 	 105 	 73 	 73 	 105
1 	 1 	 1 	 1 	 1

I 	 i 	 I 	 I 	 i
73 	 105 	 73 	 73 	 105
1 	 1 	 1 	 1 	 1

? 	 ? 	 ? 	 ? 	 ?
8491 	 229 	 8491 	 197 	 229
1 	 1 	 1 	 1 	 1

? 	 ? 	 ? 	 ? 	 ?
963 	 963 	 931 	 931 	 963
1 	 1 	 1 	 1 	 1

? 	 ? 	 ? 	 ? 	 ?
962 	 962 	 931 	 931 	 963
1 	 1 	 1 	 1 	 1

? 	 ? 	 ? 	 ? 	 ?
963 	 963 	 931 	 931 	 963
1 	 1 	 1 	 1 	 1

z 	 z 	 Z 	 Z 	 z
122 	 122 	 90 	 90 	 122
1 	 1 	 1 	 1 	 1

I've added A, space and z.

As I said some characters have ill-defined upper and lower case 
conversions, even if some aren't as esoteric as I'd thought.

In English however the conversions are perfectly well defined for A-Z 
and a-z, while they are not meaningful for characters such as space, and 
for digits.

In English such conversions are immensely useful, and it is invaluable 
for many purposes to have upper and lower case interchangeable (for 
example, you don't have separate sections in a dictionary for letters 
starting with A and those starting with a).

So it it perfectly possible to have case conversion defined for English, 
while other alphabets can do what they like.

It is a little ridiculous however to have over two thousand distinct 
files all with the lower-case normalised name of "harry_potter".

What were we talking about again? Oh yes, belittling me because I work 
with Windows!

---------------
tab="	"

def ord1(c):	return ord(c[0])

def showcases(c):
	print (c,tab,c.lower(),tab,c.upper(),tab,c.lower().upper(),tab,
	c.upper().lower())

def showcases_ord(c):
	print (ord1(c),tab,ord1(c.lower()),tab,ord1(c.upper()),tab,
	ord1(c.lower().upper()),tab,ord1(c.upper().lower()))

def showcases_len(c):
	print (len(c),tab,len(c.lower()),tab,len(c.upper()),tab,
	len(c.lower().upper()),	tab,len(c.upper().lower()))

s="A ????iI????z"

print ("Org	L	U	L->U	U->L")

for c in s:
	showcases(c)
	showcases_ord(c)
	showcases_len(c)
	print()

-- 
Bartc


From rosuav at gmail.com  Thu Dec  8 18:58:36 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 10:58:36 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <87lgvqnhuz.fsf@elektro.pacujo.net>
References: <o21u2p$co1$1@dont-email.me> <o224o1$5d2$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <mailman.159.1481214881.2253.python-list@python.org>
 <eau1f2Fr2o8U1@mid.individual.net> <87lgvqnhuz.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmpsD4dBqVF8pb6hwOe_iPyTxMS5NuVcb0iur=2Wxk+jjA@mail.gmail.com>

On Fri, Dec 9, 2016 at 10:03 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> I find it a bit annoying that "/" cannot be used in Linux filenames.
> Slashes have very common everyday uses and it's a pity they are
> reserved. Instead, ASCII control characters are allowed, which is of no
> use.

How would you do otherwise? How do you separate directories in a path?
Remember, it needs to be easy to type, and easy to read. You could
pick some other keyboard character, but you won't really gain much.

ChrisA


From torriem at gmail.com  Thu Dec  8 19:00:40 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 17:00:40 -0700
Subject: Linux terminals vs Windows consoles - was Re: python 2.7.12 on
 Linux behaving differently than on Windows
In-Reply-To: <eau2kqFrbefU1@mid.individual.net>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me> <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <57e3aa81-f498-69a3-8dbe-d6fa3ada126c@gmail.com>
 <mailman.131.1481129628.2253.python-list@python.org>
 <o29ip0$g0v$1@dont-email.me> <easg5gFfa6vU1@mid.individual.net>
 <Jzc2A.281652$kM7.92263@fx44.am4>
 <f5073955-8ebf-8675-726d-e80208a62a5e@gmail.com>
 <423687de-4ce8-4e48-fb59-eff127b132d6@gmail.com>
 <mailman.165.1481216929.2253.python-list@python.org>
 <eau2kqFrbefU1@mid.individual.net>
Message-ID: <5c5508f2-55ed-c478-8378-ec2af1729389@gmail.com>

On 12/08/2016 02:46 PM, Gregory Ewing wrote:
> Michael Torrie wrote:
>> For example, on Macs,
>> control-key is not normally used, but rather the Command-key (the apple
>> key) which happens to be where the Alt key is on our PC keyboards.
> 
> Actually, Alt is usually mapped to Option on a Mac. The Mac
> Command key corresponds the "Windows" or "Meta" key on other
> keyboards, I think (but is used in a very different way).

I was talking about position on the keyboard.  But yes, good point.


From steve+python at pearwood.info  Thu Dec  8 19:00:47 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 09 Dec 2016 11:00:47 +1100
Subject: Name resolution and the (wrong?) LEGB rule
References: <58497384.9020900@oa-cagliari.inaf.it>
 <mailman.155.1481209690.2253.python-list@python.org>
Message-ID: <5849f431$0$1614$c3e8da3$5496439d@news.astraweb.com>

On Fri, 9 Dec 2016 01:51 am, Marco Buttu wrote:

> Sometimes the Python name resolution is explained using a LEGB rule.
> For instance, in [1] (I think also the Learning Python book gives the
> same):
> 
> "if a particular name:object mapping cannot be found in the local
> namespaces, the namespaces of the enclosed scope are being searched
> next. If the search in the enclosed scope is unsuccessful, too, Python
> moves on to the global namespace, and eventually, it will search the
> built-in namespace (side note: if a name cannot found in any of the
> namespaces, a NameError will is raised)."
> 
> AFAIK, Python has static scoping: the scope for a name is given during
> the bytecode compilation. This means that before executing the program,
> Python already know in which namespace to look for an object. So, it
> seems to me that the LEGB rule is wrong, and this is what actually
> happens:
> 
> * local name (LOAD_FAST instruction): Python looks (only) in the local
> namespace, and if it does not find the name, then it raises an
> UnboundLocalError
> 
> * global name (LOAD_GLOBAL): at first Python looks in the globals(), and
> in case the name is not there, it looks in the builtins namespace; if
> the name is neither in the global nor in the builtin namespace, it
> raises a NameError
> 
> * enclosing scope (LOAD_DEREF): there is a closure, and Python looks for
> the name in the enclosing namespace
> 
> Is that right, or am I missing something? Thanks, Marco

You are partly right, but you are also missing a few things.

(1) Technically, the LOAD_* byte codes are implementation details, and there
is no requirement for Python interpreters to use them. In fact, Jython
being built on the JVM and IronPython being built on the .Net runtime
cannot use them. Even a C-based interpreter is not required to implement
name resolution in the same way. 

(2) There are some odd corner cases in Python 2 where you use exec() or
import * inside a function to create local variables, where the CPython
interpreter no longer uses LOAD_FAST to resolve locals. (I'm going by
memory here, so I may have some of the details wrong.) You can experiment
with that if you like, but the behaviour has changed in Python 3 so its no
longer relevant except as a historical detail for those stuck on Python 2.

(To be precise, import * is forbidden inside functions, and exec() must take
an explicit namespace argument rather than implicitly applying to the local
function scope.)


(3) However, the most important case you've missed is code executed inside a
class namespace during class creation time:


class C(object):

    a = 1

    def test():  # intentionally no self
        b = 2
        return a + b

    c = 3
    d = test() + c



Don't forget classes nested inside classes...


I consider the LEGB rule to be a statement of intent, describing how a
Python implementation is expected to behave, *as if* it were following the
LEGB rule, not necessarily a statement of the implementation of any
specific Python implementation.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From plzr at mac.com  Thu Dec  8 19:19:33 2016
From: plzr at mac.com (3dB)
Date: Thu, 8 Dec 2016 16:19:33 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <mailman.181.1481232517.2253.python-list@python.org>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
 <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
 <a1a781fd-8bd6-a6e7-ca96-3e2a11c9eead@gmail.com>
 <mailman.181.1481232517.2253.python-list@python.org>
Message-ID: <8891684e-b35c-4530-bfb1-9393413019bb@googlegroups.com>

> > pip install --user virtualenv
> > 
> > It will wind up in $HOME/.local/lib/python2.7/site-packages (or similar) I
> > believe.

Michael, Skip, 

I've noticed that Python is in the highest level of the drive, HDD/Library/Python as opposed to usr/Library. Does that present a problem? 

Also noticed that Python folder contains both versions 2.6 and 2.7.


From plzr at mac.com  Thu Dec  8 19:22:35 2016
From: plzr at mac.com (3dB)
Date: Thu, 8 Dec 2016 16:22:35 -0800 (PST)
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <mailman.180.1481232428.2253.python-list@python.org>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
 <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
 <mailman.180.1481232428.2253.python-list@python.org>
Message-ID: <8edc2383-5ccc-4ba8-8c99-b5ca6dd08a4d@googlegroups.com>

> pip install --user virtualenv
> 
> It will wind up in $HOME/.local/lib/python2.7/site-packages (or similar) I
> believe.

Skip, thanks, that worked for virtualenv.

I'll now try sudo for Speechrecognition.


From torriem at gmail.com  Thu Dec  8 19:23:03 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 17:23:03 -0700
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2cpof$r89$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
Message-ID: <21b70d50-bb73-7de7-8f9b-59d540137de4@gmail.com>

On 12/08/2016 04:19 PM, BartC wrote:
> What were we talking about again? Oh yes, belittling me because I work 
> with Windows!

Yes it's kind of getting that way, which means this conversation is at
an end.  I don't think Chris or Steven or anyone else who has gone round
and round with you has meant to demean you personally.

Some of the responses have been a bit strong, because your own arguments
have come off rather harsh--something along the lines of you can't
believe anyone would actually use Linux, let alone like it because it's
so backwards compared to the way you are used to, in your own opinion.

None of this would matter much, but you apparently want to port your
programs to run under Linux, which is going to be a really rough
experience without a little broad-mindedness on your part.

And now for something completely different, and to move the thread back
on topic, I think everyone would be well served to watch this little
educational and informational Python video:

https://www.youtube.com/watch?v=kQFKtI6gn9Y


From torriem at gmail.com  Thu Dec  8 19:29:53 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 17:29:53 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <8891684e-b35c-4530-bfb1-9393413019bb@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
 <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
 <a1a781fd-8bd6-a6e7-ca96-3e2a11c9eead@gmail.com>
 <mailman.181.1481232517.2253.python-list@python.org>
 <8891684e-b35c-4530-bfb1-9393413019bb@googlegroups.com>
Message-ID: <c92e38ad-525a-bf64-e961-d2c0fe55c51b@gmail.com>

On 12/08/2016 05:19 PM, 3dB wrote:
>>> pip install --user virtualenv
>>> 
>>> It will wind up in $HOME/.local/lib/python2.7/site-packages (or
>>> similar) I believe.
> 
> Michael, Skip,
> 
> I've noticed that Python is in the highest level of the drive,
> HDD/Library/Python as opposed to usr/Library. Does that present a
> problem?
> 
> Also noticed that Python folder contains both versions 2.6 and 2.7.

OS X ships with both Python 2.6 and 2.7 apparently. The default is
probably 2.7.

System-wide libraries and other data are installed to /Library, which is
not typically writable to a normal user.  There are versions of all of
these directories installed in your user home directory, which are
writable to you and don't require any special privileges.  Skip was
saying if you can install to the directories in your home directory,
($HOME/Library/whatever instead of /Library/whatever) that is best since
it doesn't modify the system at all and so it won't get overwritten by
system updates or re-installs.  If you can run without root privileges
(without sudo) and use directories in your own home directory for python
modules you should.  I don't use OS X, so beyond this I am not of much
help.  Perhaps other Mac users will see this thread and pipe up.


From torriem at gmail.com  Thu Dec  8 19:42:50 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 8 Dec 2016 17:42:50 -0700
Subject: MacOSX SpeechRecognition installation problems
In-Reply-To: <8edc2383-5ccc-4ba8-8c99-b5ca6dd08a4d@googlegroups.com>
References: <98ce6c34-3df9-4d4c-9102-7d284945fe49@googlegroups.com>
 <e61adfae-c43d-e2b9-3959-001721038fc0@gmail.com>
 <7627C602-3E32-4D12-BBA9-51B03FD63DBB@gmail.com>
 <mailman.153.1481188555.2253.python-list@python.org>
 <2e55264a-f2cd-4dce-8f64-b66aac6f5a04@googlegroups.com>
 <6635ec1b-5892-03da-95ef-51e41c34ba9c@gmail.com>
 <mailman.174.1481222097.2253.python-list@python.org>
 <2742772e-4352-4b53-9385-6a5a1313366e@googlegroups.com>
 <a88fa7bf-170f-9b82-9055-4c8d5beac6fb@gmail.com>
 <CANc-5UwGKB_8vTURYO-Z2vaVhtTO_3q3BqAnOJ14nF4DO9rJJg@mail.gmail.com>
 <mailman.180.1481232428.2253.python-list@python.org>
 <8edc2383-5ccc-4ba8-8c99-b5ca6dd08a4d@googlegroups.com>
Message-ID: <a6a29316-5b8f-63d9-a3d7-c2c34fe112bf@gmail.com>

On 12/08/2016 05:22 PM, 3dB wrote:
>> pip install --user virtualenv
>>
>> It will wind up in $HOME/.local/lib/python2.7/site-packages (or similar) I
>> believe.
> 
> Skip, thanks, that worked for virtualenv.
> 
> I'll now try sudo for Speechrecognition.

You should be able to install speach recognition to the local virtualenv
also.  Shouldn't need sudo now.  I think, anyway.



From rosuav at gmail.com  Thu Dec  8 19:55:44 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 11:55:44 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2cpof$r89$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me> <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
Message-ID: <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>

On Fri, Dec 9, 2016 at 10:19 AM, BartC <bc at freeuk.com> wrote:
> I get this (although I suspect Thunderbird will screw up the tabs); the code I used follows at the end:

Actually it came through just fine. Although, point of note: the case
conversions of individual characters are not the same as the case
conversions of the whole string.

> As I said some characters have ill-defined upper and lower case conversions,
> even if some aren't as esoteric as I'd thought.

Can you show me any character with ill-defined conversions?

> In English however the conversions are perfectly well defined for A-Z and
> a-z, while they are not meaningful for characters such as space, and for
> digits.

Correct; caseless characters don't change. There's no concept of
"upper-case 5", and no, it isn't "%", despite people who think that
"upper-case" means "hold shift" :)

> In English such conversions are immensely useful, and it is invaluable for
> many purposes to have upper and lower case interchangeable (for example, you
> don't have separate sections in a dictionary for letters starting with A and
> those starting with a).

That's not quite the same; that's a *collation order*. And some
dictionaries will sort them all in together, but others will put the
uppercase before the lowercase (ie sorting "AaBbCcDd"). It's not that
they're considered identical; it's that they're placed near each other
in the sort. In fact, it's possible to have collations without
conversions - for instance, it's common for McFoo and MacFoo to be
sorted together in a list of names.

> So it it perfectly possible to have case conversion defined for English,
> while other alphabets can do what they like.

Aaaaand there we have it. Not only do you assume that English is the
only thing that matters, you're happy to give the finger to everyone
else on the planet.

> It is a little ridiculous however to have over two thousand distinct files
> all with the lower-case normalised name of "harry_potter".

It's also ridiculous to have hundreds of files with unique two-letter
names, but I'm sure someone has done it. The file system shouldn't
stop you, any more than it should stop you from having "swimmer" and
"swirnrner" in the same directory (in some fonts, they're
indistinguishable).

> What were we talking about again? Oh yes, belittling me because I work with
> Windows!

Or because you don't understand anything outside of what you have
worked with, and assume that anything you don't understand must be
inferior. It's not a problem to not know everything, but you
repeatedly assert that other ways of doing things are "wrong", without
acknowledging that these ways have worked for forty years and are
strongly *preferred* by myriad people around the world. I grew up on
OS/2, using codepage 437 and case insensitive file systems, and there
was no way for me to adequately work with other Latin-script
languages, much less other European languages (Russian, Greek), and
certainly I had no way of working with non-alphabetic languages
(Chinese, Japanese). Nor right-to-left languages (Hebrew, Arabic). And
I didn't know anything about the Unix security model, with different
processes being allowed to do different things. Today, the world is a
single place, not lots of separate places that sorta kinda maybe
grudgingly talk to each other. We aren't divided according to "people
who use IBMs" and "people who use DECs". We shouldn't be divided
according to "people who use CP-850" and "people who use
Windows-1253". The tools we have today are capable of serving everyone
properly. Let's use them.

ChrisA


From bc at freeuk.com  Thu Dec  8 20:34:34 2016
From: bc at freeuk.com (BartC)
Date: Fri, 9 Dec 2016 01:34:34 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <mailman.188.1481244947.2253.python-list@python.org>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
Message-ID: <o2d1ku$gfo$1@dont-email.me>

On 09/12/2016 00:55, Chris Angelico wrote:
> On Fri, Dec 9, 2016 at 10:19 AM, BartC <bc at freeuk.com> wrote:

>> So it it perfectly possible to have case conversion defined for English,
>> while other alphabets can do what they like.
>
> Aaaaand there we have it. Not only do you assume that English is the
> only thing that matters, you're happy to give the finger to everyone
> else on the planet.

I haven't given the finger to anybody. I'm not an expert on all these 
other languages, but I don't want to tell them what to do either. /They/ 
decide what meaning upper/lower has, if any.

Or maybe we can just do away with it altogether. (What are Python's 
.lower/.upper used for, after all? With any such functions, you are 
guaranteed to find a Unicode sequence which is going to give trouble. So 
take them out so as not to offend anyone!)

It does seem however as though users of other languages are telling /me/ 
how I should deal with English, which is what I spend nearly 100% of my 
life dealing with (those language lessons not having worked out...).

>
>> It is a little ridiculous however to have over two thousand distinct files
>> all with the lower-case normalised name of "harry_potter".
>
> It's also ridiculous to have hundreds of files with unique two-letter
> names, but I'm sure someone has done it. The file system shouldn't
> stop you

With a case-sensitive file system, how do you search only for 'harry', 
not knowing what combinations of upper and lower case have been used? 
(It's a good thing Google search isn't case sensitive!)

> What were we talking about again? Oh yes, belittling me because I work with
>> Windows!
>
> Or because you don't understand anything outside of what you have
> worked with, and assume that anything you don't understand must be
> inferior. It's not a problem to not know everything, but you
> repeatedly assert that other ways of doing things are "wrong", without
> acknowledging that these ways have worked for forty years and are
> strongly *preferred* by myriad people around the world. I grew up on
> OS/2, using codepage 437 and case insensitive file systems, and there
> was no way for me to adequately work with other Latin-script
> languages,

I developed applications that needed to work in French, German and 
Dutch, apart from English. That might have been long enough ago that I 
may have had to provide some of the fonts (both bitmap and vector), as 
well as implement suitable keyboard layouts used with digitising 
tablets. /And/ provide support in the scripting languages that went with 
them.

Not particularly demanding in terms of special or quirky characters (or 
word-order issues when constructing messages for translation) but it's 
not quite the same as my assuming everything was 7-bit ASCII and in English.

  much less other European languages (Russian, Greek), and
> certainly I had no way of working with non-alphabetic languages
> (Chinese, Japanese). Nor right-to-left languages (Hebrew, Arabic).

Did you really need to work with all those languages, or is this a 
generic 'I'.

-- 
Bartc


From nathan.ernst at gmail.com  Thu Dec  8 20:43:41 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Thu, 8 Dec 2016 19:43:41 -0600
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2d1ku$gfo$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
Message-ID: <CAPjBxg_7N5aioNMLNA4FfiLy5_225eqhqvewAtR6xaEML=SONw@mail.gmail.com>

With a case-sensitive file system, how do you search only for 'harry', not
knowing what combinations of upper and lower case have been used? (It's a
good thing Google search isn't case sensitive!)

On Linux, I'd do "find . -iname harry". A lot, but not all, of the tools
usually have options to ignore case. I don't know how this works with
unicode, as I don't normally have to deal with non-ASCII, either. Probably
the only time I see unicode is when I see files with symbols for currencies
instead of the ISO-3 currency code (I work in finance) and occasionally in
company names.

Regards,
Nate

On Thu, Dec 8, 2016 at 7:34 PM, BartC <bc at freeuk.com> wrote:

> On 09/12/2016 00:55, Chris Angelico wrote:
>
>> On Fri, Dec 9, 2016 at 10:19 AM, BartC <bc at freeuk.com> wrote:
>>
>
> So it it perfectly possible to have case conversion defined for English,
>>> while other alphabets can do what they like.
>>>
>>
>> Aaaaand there we have it. Not only do you assume that English is the
>> only thing that matters, you're happy to give the finger to everyone
>> else on the planet.
>>
>
> I haven't given the finger to anybody. I'm not an expert on all these
> other languages, but I don't want to tell them what to do either. /They/
> decide what meaning upper/lower has, if any.
>
> Or maybe we can just do away with it altogether. (What are Python's
> .lower/.upper used for, after all? With any such functions, you are
> guaranteed to find a Unicode sequence which is going to give trouble. So
> take them out so as not to offend anyone!)
>
> It does seem however as though users of other languages are telling /me/
> how I should deal with English, which is what I spend nearly 100% of my
> life dealing with (those language lessons not having worked out...).
>
>
>> It is a little ridiculous however to have over two thousand distinct files
>>> all with the lower-case normalised name of "harry_potter".
>>>
>>
>> It's also ridiculous to have hundreds of files with unique two-letter
>> names, but I'm sure someone has done it. The file system shouldn't
>> stop you
>>
>
> With a case-sensitive file system, how do you search only for 'harry', not
> knowing what combinations of upper and lower case have been used? (It's a
> good thing Google search isn't case sensitive!)
>
> What were we talking about again? Oh yes, belittling me because I work with
>>
>>> Windows!
>>>
>>
>> Or because you don't understand anything outside of what you have
>> worked with, and assume that anything you don't understand must be
>> inferior. It's not a problem to not know everything, but you
>> repeatedly assert that other ways of doing things are "wrong", without
>> acknowledging that these ways have worked for forty years and are
>> strongly *preferred* by myriad people around the world. I grew up on
>> OS/2, using codepage 437 and case insensitive file systems, and there
>> was no way for me to adequately work with other Latin-script
>> languages,
>>
>
> I developed applications that needed to work in French, German and Dutch,
> apart from English. That might have been long enough ago that I may have
> had to provide some of the fonts (both bitmap and vector), as well as
> implement suitable keyboard layouts used with digitising tablets. /And/
> provide support in the scripting languages that went with them.
>
> Not particularly demanding in terms of special or quirky characters (or
> word-order issues when constructing messages for translation) but it's not
> quite the same as my assuming everything was 7-bit ASCII and in English.
>
>  much less other European languages (Russian, Greek), and
>
>> certainly I had no way of working with non-alphabetic languages
>> (Chinese, Japanese). Nor right-to-left languages (Hebrew, Arabic).
>>
>
> Did you really need to work with all those languages, or is this a generic
> 'I'.
>
>
> --
> Bartc
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From rosuav at gmail.com  Thu Dec  8 21:52:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 13:52:19 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <o2d1ku$gfo$1@dont-email.me>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
Message-ID: <CAPTjJmqmdXRPwRJh26FvV_=COauk9Am7LVA0_+DjQy48Zy8XdQ@mail.gmail.com>

On Fri, Dec 9, 2016 at 12:34 PM, BartC <bc at freeuk.com> wrote:
> With a case-sensitive file system, how do you search only for 'harry', not
> knowing what combinations of upper and lower case have been used? (It's a
> good thing Google search isn't case sensitive!)

This is handled by "case correlations", which aren't quite the same as
case conversions. In Python, that's the .casefold() method. You
casefold the string "harry", and then casefold every file name that
might potentially match, and see if they become the same. Taking my
original example string:

>>> "????????".casefold()
'ssss?i?????'

Any string that casefolds to that same string should be considered a
match. This does NOT stop you from having multiple such files in the
directory.

>>  much less other European languages (Russian, Greek), and
>> certainly I had no way of working with non-alphabetic languages
>> (Chinese, Japanese). Nor right-to-left languages (Hebrew, Arabic).
>
>
> Did you really need to work with all those languages, or is this a generic
> 'I'.

In the 90s? No, I didn't, otherwise I'd have had a lot of trouble.
Today? Yep. I can, and I do. Not as much as I work with English, but I
definitely do support them. Took a good bit of work to make some of
the RTL text behaviours the way I wanted them, but everything else was
pretty smooth.

ChrisA


From vek.m1234 at gmail.com  Thu Dec  8 23:13:09 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Fri, 09 Dec 2016 09:43:09 +0530
Subject: Working around multiple files in a folder
References: <a8b89adf-c529-4f01-ad4b-362fb087a489@googlegroups.com>
 <o0voc0$qg1$1@blaine.gmane.org>
 <mailman.25.1479763212.3146.python-list@python.org>
Message-ID: <o2daub$uek$1@dont-email.me>

Emile van Sebille wrote:

> On 11/21/2016 11:27 AM, subhabangalore at gmail.com wrote:
>> I have a python script where I am trying to read from a list of files
>> in a folder and trying to process something. As I try to take out the
>> output I am presently appending to a list.
>>
>> But I am trying to write the result of individual files in individual
>> list or files.
>>
>> The script is as follows:
>>
>> import glob
>> def speed_try():
>>      #OPENING THE DICTIONARY
>>      a4=open("/python27/Dictionaryfile","r").read()
>>      #CONVERTING DICTIONARY INTO WORDS
>>      a5=a4.lower().split()
>>      list1=[]
>>      for filename in glob.glob('/Python27/*.txt'):
>>          a1=open(filename,"r").read()
>>          a2=a1.lower()
>>          a3=a2.split()
>>          for word in a3:
>>              if word in a5:
>>                  a6=a5.index(word)
>>                  a7=a6+1
>>                  a8=a5[a7]
>>                  a9=word+"/"+a8
>>                  list1.append(a9)
>>              elif word not in a5:
>>                  list1.append(word)
>>              else:
>>                  print "None"
>>
>>      x1=list1
>>      x2=" ".join(x1)
>>      print x2
>>
>> Till now, I have tried to experiment over the following solutions:
>>
>> a) def speed_try():
>>        #OPENING THE DICTIONARY
>>        a4=open("/python27/Dictionaryfile","r").read()
>>        #CONVERTING DICTIONARY INTO WORDS
>>        a5=a4.lower().split()
>>        list1=[]
>>        for filename in glob.glob('/Python27/*.txt'):
>>           a1=open(filename,"r").read()
>>           a2=a1.lower()
>>           a3=a2.split()
>>            list1.append(a3)
>>
>>
>>      x1=list1
>>      print x1
>>
>> Looks very close but I am unable to fit the if...elif...else part.
>>
>> b) import glob
>> def multi_filehandle():
>>      list_of_files = glob.glob('/Python27/*.txt')
>>      for file_name in list_of_files:
>>          FI = open(file_name, 'r')
>>          FI1=FI.read().split()
>>          FO = open(file_name.replace('txt', 'out'), 'w')
>>          for line in FI:
> 
> at this point, there's nothing left to be read from FI having been
> fully drained to populate FI1 -- maybe you want to loop over FI1
> instead?
> 
> Emile
> 
> 
>>              FO.write(line)
>>
>>          FI.close()
>>          FO.close()
>>
>> I could write output but failing to do processing of the files
>> between opening and writing.
>>
>> I am trying to get examples from fileinput.
>>
>> If anyone of the learned members may kindly suggest how may I
>> proceed.
>>
>> I am using Python2.x on MS-Windows.
>>
>> The practices are scripts and not formal codes so I have not followed
>> style guides.
>>
>> Apology for any indentation error.
>>
>> Thanking in advance.
>>
>>
*goggles in shock* that was painful to read! The best I could make of it 
was, he's trying to match words in some.txt against a dictionary.

(OP) should not code like a horror movie. Try to avoid dismembering your 
variable names and numbering the body parts like a serial killer. 
Instead try to pick names that matter. Use functions to hide the gory 
complexity..

Maybe like this (I dunno what you are doing)..

dict_file = '/python27/Dictionaryfile'
txt_file_path = '/Python27/*.txt'


def open_file(fname, mode='r'):
    lines = open(fname, mode).read()
    words = lines.lower().split()
    return words

def get_files(path):
    file_list = []
    for fname in glob.glob(path):
        file_list.append(fname)

    return file_list


word_list = open_file(dict_file, mode)
file_list = get_files(txt_file_path)


for fname in file_list:
 do something and stick this in a func when you know what you were doing

Otherwise you'll get minimal help online because your program is 
unreadable.. in-spite of the frivolous comments.


From vek.m1234 at gmail.com  Thu Dec  8 23:29:12 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Fri, 09 Dec 2016 09:59:12 +0530
Subject: Simple code and suggestion
References: <CALM1kKWgjo6fXPZ8tjfaEX5gL9SiJBVfsYKzqzt-7Hd+5X8prQ@mail.gmail.com>
 <mailman.12.1480512384.2387.python-list@python.org>
Message-ID: <o2dbs9$p1$1@dont-email.me>

g thakuri wrote:

> Dear Python friends,
> 
> I have a simple question , need your suggestion the same
> 
> I would want to avoid using multiple split in the below code , what
> options do we have before tokenising the line?, may be validate the
> first line any other ideas
> 
>  cmd = 'utility   %s' % (file)
>  out, err, exitcode = command_runner(cmd)
>  data = stdout.strip().split('\n')[0].split()[5][:-2]
> 
> Love,

import re
? regex/pattern matching module..


From steve+python at pearwood.info  Fri Dec  9 00:07:42 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 09 Dec 2016 16:07:42 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <mailman.159.1481214881.2253.python-list@python.org>
 <eau1f2Fr2o8U1@mid.individual.net> <87lgvqnhuz.fsf@elektro.pacujo.net>
 <CAPTjJmpsD4dBqVF8pb6hwOe_iPyTxMS5NuVcb0iur=2Wxk+jjA@mail.gmail.com>
 <v02k4ch82emcg3eh9uil6g9to2punl7fl4@4ax.com>
 <mailman.189.1481247142.2253.python-list@python.org>
Message-ID: <584a3c20$0$1586$c3e8da3$5496439d@news.astraweb.com>

On Fri, 9 Dec 2016 12:32 pm, Dennis Lee Bieber wrote:

> There is always VMS:
> 
> volume:[dir.sub.subsub]fn.ext;ver
> 
> (where volume could be a hardware device or a logical name -- which is NOT
> the same as an environment variable; only other place I've seen something
> similar is the Amiga -- where DH0: [or HD0: later] was the physical hard
> drive 0, SYS: was a logical name for the drive the system was loaded off,
> and OS: could have been the volume name assigned when formatting; and
> asking for files by volume name rather than drive name would result in a
> prompt to insert the named volume into ANY drive slot, and the OS would
> figure out where it was; the only time one was asked to insert media into
> a specific drive is when there was an open file on the drive -- since open
> files went through drive specific file handler instances)

Sounds similar to classic Mac OS, where you had:


    volume:path


where the path uses : as directory separators. Removable media (floppies,
removable hard drives) could be unplugged and re-inserted into any physical
drive, and the OS would track where the media was and whether it was
physically mounted or not.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From random832 at fastmail.com  Fri Dec  9 00:32:35 2016
From: random832 at fastmail.com (Random832)
Date: Fri, 09 Dec 2016 00:32:35 -0500
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
Message-ID: <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>

On Thu, Dec 8, 2016, at 20:38, Dennis Lee Bieber wrote:
> On Thu, 08 Dec 2016 10:37:27 -0500, Random832 <random832 at fastmail.com>
> declaimed the following:
> >There are other issues, like needing a way to do Windows' version of
> >wildcard parsing with all its quirks, or at least some of its quirks -
> >"*.*" for all files and "*." for files not containing any dot being the
> >most commonly used in the real world.
> >
> 	In the original 8.3 scheme -- no files "contained" a dot

Yes, but they do now, and the compatibility quirks persist. At some
point in the process *?. are translated to other reserved characters <>"
in order to allow the same native functions to perform both the 'quirks
mode' and more rigorous logic.


From marko at pacujo.net  Fri Dec  9 00:44:26 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 07:44:26 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o21u2p$co1$1@dont-email.me>
 <5844a5d9$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o23jo4$30k$1@dont-email.me>
 <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <mailman.44.1480960040.2253.python-list@python.org>
 <o24cr9$1ri$1@dont-email.me>
 <976c4c1brg51dfbsaf1a7jl20mcn214j6j@4ax.com>
 <mailman.69.1480990884.2253.python-list@python.org>
 <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28ifv$ros$1@blaine.gmane.org>
 <1481212651.866944.812740609.27AA4416@webmail.messagingengine.com>
 <o2c22j$pin$1@blaine.gmane.org>
 <mailman.159.1481214881.2253.python-list@python.org>
 <eau1f2Fr2o8U1@mid.individual.net> <87lgvqnhuz.fsf@elektro.pacujo.net>
 <CAPTjJmpsD4dBqVF8pb6hwOe_iPyTxMS5NuVcb0iur=2Wxk+jjA@mail.gmail.com>
 <mailman.183.1481241525.2253.python-list@python.org>
Message-ID: <87fulxodvp.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Fri, Dec 9, 2016 at 10:03 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> I find it a bit annoying that "/" cannot be used in Linux filenames.
>> Slashes have very common everyday uses and it's a pity they are
>> reserved. Instead, ASCII control characters are allowed, which is of
>> no use.
>
> How would you do otherwise? How do you separate directories in a path?
> Remember, it needs to be easy to type, and easy to read. You could
> pick some other keyboard character, but you won't really gain much.

There are numerous ways. For example:

   [ "firmware", "CP/M", "2016-12-09" ]
   path("firmware", "CP/M", "2016-12-09")
   file:///firmware/CP%2fM/2016-12-09


Marko


From marko at pacujo.net  Fri Dec  9 00:52:33 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 07:52:33 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
Message-ID: <87bmwlodi6.fsf@elektro.pacujo.net>

Random832 <random832 at fastmail.com>:

> On Thu, Dec 8, 2016, at 20:38, Dennis Lee Bieber wrote:
>> 	In the original 8.3 scheme -- no files "contained" a dot
>
> Yes, but they do now, and the compatibility quirks persist.

When porting a Python program to Windows, I noticed the filename "aux"
is not allowed in Windows. I suspect it's the same with "lst", "prn",
"con" and what not.

In Linux, "." and ".." are taboo. So is "".


Marko


From steve+python at pearwood.info  Fri Dec  9 02:41:51 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 09 Dec 2016 18:41:51 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
 <CAPTjJmqmdXRPwRJh26FvV_=COauk9Am7LVA0_+DjQy48Zy8XdQ@mail.gmail.com>
 <mailman.192.1481251948.2253.python-list@python.org>
Message-ID: <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>

On Fri, 9 Dec 2016 01:52 pm, Chris Angelico wrote:

> On Fri, Dec 9, 2016 at 12:34 PM, BartC <bc at freeuk.com> wrote:
>> With a case-sensitive file system, how do you search only for 'harry',
>> not knowing what combinations of upper and lower case have been used?
>> (It's a good thing Google search isn't case sensitive!)
> 
> This is handled by "case correlations", which aren't quite the same as
> case conversions. 

I think you mean to say, this *should* be handled by case correlations.

I expect that there is a lot of software on the planet that just blindly
converts the strings to both lower- or both uppercase and then checks
equality. And even more that assumes ASCII case conversions and just does
bit-twiddling to compare letters.

(By the way, case correlation... I've never come across that term before,
and googling doesn't find anything useful. Are you sure that's the right
term?)


> In Python, that's the .casefold() method. You 
> casefold the string "harry", and then casefold every file name that
> might potentially match, and see if they become the same. Taking my
> original example string:
> 
>>>> "????????".casefold()
> 'ssss?i?????'

Your example string is a good demonstration of mojibake. Or possibly a *bad*
demonstration of mojibake, since I cannot imagine any real words that would
generate that via encoding/decoding into the wrong character set :-)

I'm not really sure what point you were trying to make with Bart. Globbing
doesn't support case-insensitive matches on any platform I know of, so I
don't think this is really relevant.


> Any string that casefolds to that same string should be considered a
> match. This does NOT stop you from having multiple such files in the
> directory.

If you want to support multilingual string comparisons, you have to do more
than just casefold(). You need to normalise the strings, otherwise 'caf?'
and 'cafe?' will be treated as distinct.

Frankly, I think that Apple HFS+ is the only modern file system that gets
Unicode right. Not only does it restrict file systems to valid UTF-8
sequences, but it forces them to a canonical form to avoid the ? e? gotcha,
and treats file names as case preserving but case insensitive.

Lastly, there's one last fly in the ointment for multilingual case
comparisons: Turkish i. Unfortunately, there's no clean way to do case
comparisons that works for "any arbitrary language".

Turkish, and one or two other languages, want dotless and dotted I to be
treated as distinct: ?I go together, and i? go together. But other
languages want iI to go together, meaning that the standard case
conversions are lossy:

py> '?Ii?'.lower()
'?iii'
py> '?Ii?'.upper()
'III?'


Maybe it would have been better if the standard had kept ?? together, and
iI, so at least the case conversion was lossless. Alas, too late now.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Fri Dec  9 03:26:02 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 09 Dec 2016 19:26:02 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
 <87bmwlodi6.fsf@elektro.pacujo.net>
Message-ID: <584a6a9b$0$1621$c3e8da3$5496439d@news.astraweb.com>

On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:

> Random832 <random832 at fastmail.com>:
> 
>> On Thu, Dec 8, 2016, at 20:38, Dennis Lee Bieber wrote:
>>> In the original 8.3 scheme -- no files "contained" a dot
>>
>> Yes, but they do now, and the compatibility quirks persist.
> 
> When porting a Python program to Windows, I noticed the filename "aux"
> is not allowed in Windows. I suspect it's the same with "lst", "prn",
> "con" and what not.
> 
> In Linux, "." and ".." are taboo. 

No that's incorrect. It isn't that . and .. are forbidden, but they are
reserved: every single directory in Unix file systems have a . and ..
directory entry. So they are legitimate directory names -- they're just not
names you can use for your own files, as they are already in use.

> So is "".

That's one way of looking at things... I'm not sure that the empty string
counts as a file name. Its more of the LACK of a file name.

Besides, that would be ambiguous. Would "/home/steve/" mean my home
directory, or the file "" inside my home directory?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From marko at pacujo.net  Fri Dec  9 05:34:56 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 12:34:56 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
 <87bmwlodi6.fsf@elektro.pacujo.net>
 <584a6a9b$0$1621$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <877f791jcf.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:
>> In Linux, "." and ".." are taboo. 
>
> No that's incorrect. It isn't that . and .. are forbidden, but they are
> reserved: every single directory in Unix file systems have a . and ..
> directory entry. So they are legitimate directory names -- they're just not
> names you can use for your own files, as they are already in use.

Same difference.

>> So is "".
>
> That's one way of looking at things... I'm not sure that the empty
> string counts as a file name. Its more of the LACK of a file name.
>
> Besides, that would be ambiguous. Would "/home/steve/" mean my home
> directory, or the file "" inside my home directory?

If Python had been the standard shell when Unix came about, you could
have defined pathnames as lists of strings. Then, everything would be
unambigous and there wouldn't be any taboo or reserved names.

BTW, guile allows *any* characters in an identifier. Even the empty name
is a valid identifier:

   > (define #{}# "hello")
   > #{}#
   $1 = "hello"
   (symbol->string '#{}#)
   $2 = ""
   > (symbol->string '#{.}#)
   $4 = "."


Marko


From rosuav at gmail.com  Fri Dec  9 05:51:26 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 9 Dec 2016 21:51:26 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
 <CAPTjJmqmdXRPwRJh26FvV_=COauk9Am7LVA0_+DjQy48Zy8XdQ@mail.gmail.com>
 <mailman.192.1481251948.2253.python-list@python.org>
 <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmqrtOzwdELx_hwc8pXNqunaakqsT_i-8LvUEZHQN2mx0Q@mail.gmail.com>

On Fri, Dec 9, 2016 at 6:41 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Fri, 9 Dec 2016 01:52 pm, Chris Angelico wrote:
>
>> On Fri, Dec 9, 2016 at 12:34 PM, BartC <bc at freeuk.com> wrote:
>>> With a case-sensitive file system, how do you search only for 'harry',
>>> not knowing what combinations of upper and lower case have been used?
>>> (It's a good thing Google search isn't case sensitive!)
>>
>> This is handled by "case correlations", which aren't quite the same as
>> case conversions.
>
> I think you mean to say, this *should* be handled by case correlations.
>
> I expect that there is a lot of software on the planet that just blindly
> converts the strings to both lower- or both uppercase and then checks
> equality. And even more that assumes ASCII case conversions and just does
> bit-twiddling to compare letters.

This is true. However, I would consider that to be buggy software, not
flawed design of file systems.

> (By the way, case correlation... I've never come across that term before,
> and googling doesn't find anything useful. Are you sure that's the right
> term?)

Hmm, now you mention it, I'm actually not sure where I got that term
from. But it doesn't matter what you call it; the point is that there
is a "case insensitive comparison equivalency" that is not the same as
merely converting the string to upper/lower. It's allowed to be lossy;
in fact, I would start the process with NFKC or NFKD normalization, to
eliminate any problems with ligatures and such.

>> In Python, that's the .casefold() method. You
>> casefold the string "harry", and then casefold every file name that
>> might potentially match, and see if they become the same. Taking my
>> original example string:
>>
>>>>> "????????".casefold()
>> 'ssss?i?????'
>
> Your example string is a good demonstration of mojibake. Or possibly a *bad*
> demonstration of mojibake, since I cannot imagine any real words that would
> generate that via encoding/decoding into the wrong character set :-)

Heh. It's more of a "stress test" file name, picking up edge cases
from several languages. It's plausible for a file name to have one or
two of those characters in it, and for different file names to have
different selections from that set, and for a single file system to
have to cope with all of those examples.

> I'm not really sure what point you were trying to make with Bart. Globbing
> doesn't support case-insensitive matches on any platform I know of, so I
> don't think this is really relevant.

Windows does case insensitive matches. Has for as long as I've known it.

>> Any string that casefolds to that same string should be considered a
>> match. This does NOT stop you from having multiple such files in the
>> directory.
>
> If you want to support multilingual string comparisons, you have to do more
> than just casefold(). You need to normalise the strings, otherwise 'caf?'
> and 'cafe?' will be treated as distinct.

IMO you should be able to NFC normalize file names before they get
stored. The only reason Linux file systems currently can't is backward
compat - the file names might not represent text. But they are
*names*. Fundamentally, they are supposed to be meaningful. Mandating
that they be UTF-8 byte streams representing Unicode text is not
unreasonable.

> Frankly, I think that Apple HFS+ is the only modern file system that gets
> Unicode right. Not only does it restrict file systems to valid UTF-8
> sequences, but it forces them to a canonical form to avoid the ? e? gotcha,
> and treats file names as case preserving but case insensitive.

Agreed. Other file systems and operating systems should pick this up.

> Lastly, there's one last fly in the ointment for multilingual case
> comparisons: Turkish i. Unfortunately, there's no clean way to do case
> comparisons that works for "any arbitrary language".

You may notice that my original string has a couple of examples of that :)

ChrisA


From eryksun at gmail.com  Fri Dec  9 06:01:57 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 9 Dec 2016 11:01:57 +0000
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
 <CAPTjJmqmdXRPwRJh26FvV_=COauk9Am7LVA0_+DjQy48Zy8XdQ@mail.gmail.com>
 <mailman.192.1481251948.2253.python-list@python.org>
 <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CACL+1atSf8_UstdhdOeaRQ6bj4EkvZtCh_fQCLfNFBL3BYCZaA@mail.gmail.com>

On Fri, Dec 9, 2016 at 7:41 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> Frankly, I think that Apple HFS+ is the only modern file system that gets
> Unicode right. Not only does it restrict file systems to valid UTF-8
> sequences, but it forces them to a canonical form to avoid the ? e? gotcha,
> and treats file names as case preserving but case insensitive.

Windows NTFS doesn't normalize names to a canonical form. It also
allows lone surrogate codes, which is invalid UTF-16.

For case insensitive matches it converts to upper case, but the
conversion table it uses is extremely conservative. Here's a simple
function to convert a string to upper case using NT's runtime library
function RtlUpcaseUnicodeChar:

    import ctypes

    ntdll = ctypes.WinDLL('ntdll')

    def upcase(s):
        up = []
        for c in s:
            b = bytearray()
            for c in memoryview(c.encode('utf-16le')).cast('H'):
                c_up = ntdll.RtlUpcaseUnicodeChar(c)
                b += c_up.to_bytes(2, 'little')
            up.append(b.decode('utf-16le'))
        return ''.join(up)

For example:

    >>> upcase('abcd')
    'ABCD'
    >>> upcase('????')
    '????'
    >>> upcase('????????')
    '????????'

Attempting to create two files named '????????' and '????????' in the
same NTFS directory fails, as expected:

    >>> s = '????????'
    >>> open(s, 'x').close()
    >>> open(upcase(s), 'x').close()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileExistsError: [Errno 17] File exists: '????????'

Note that Windows thinks standard case conversions of this name are all unique:

    >>> open(s.upper(), 'x').close()
    >>> open(s.lower(), 'x').close()
    >>> open(s.casefold(), 'x').close()
    >>> os.listdir()
    ['ssss?i?????', 'SS?I?????', '???i?????', '????????']


From skybuck2000 at hotmail.com  Fri Dec  9 07:26:21 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Fri, 9 Dec 2016 04:26:21 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each other, 
 given a victory chance for number to number attack.
Message-ID: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>

Hello,

(This problem is probably too computationally intensive to solve with Python, though Python + Cuda could be interesting, and also Python has some interesting expressive powers, so it could be interesting to see how Python programmers might be able to express this problem with Python code, so I am going to give this Python group a try... maybe it will surprise me ! :) At least now you have a nice computational problem for those boring rainy days (when the net is down?and the offline games bore you ;) LOL :))

There are two groups of numbers which try to destroy each other:

Group X = 1,2,3,4
Group Y = 1,2,3,4

There is a 4 by 4 victory table which describes the chance of a number 
destroying another number:

Victory table =
50,  3, 80, 70
90, 60, 20, 40
30, 90, 55, 65
75, 90, 98, 60

(Currently implemented as a chance by diving it by 100 and storing as 
floating point, but since these are subtracted only from 1.0 I guess they 
can be stored as integers instead, even bytes)

This leads to the following computational problem as far as I am concerned:

Each number has an attack order permutation as follows (factorial 4 = 
1x2x3x4 = 24)

1,2,3,4    // 1
1,2,4,3    // 2
1,3,2,4    // 3
1,3,4,2    // 4
1,4,2,3    // 5
1,4,3,2    // 6
2,1,3,4    // 7
2,1,4,3    // 8
2,3,1,4    // 9
2,3,4,1    // 10
2,4,1,3    // 11
2,4,3,1    // 12
3,1,2,4    // 13
3,1,4,2    // 14
3,2,1,4    // 15
3,2,4,1    // 16
3,4,1,2    // 17
3,4,2,1    // 18
4,1,2,3    // 19
4,1,3,2    // 20
4,2,1,3    // 21
4,2,3,1    // 22
4,3,1,2    // 23
4,3,2,1   // 24

(These attack orders can be numbered from 1 to 24 or 0 to 23 and then it's 
attack order/permutation can be looked up to safe memory.)

Each number has it's own attack order and thus this leads to the following 
combinational computational problem:

All combinations of permutations in which order group X can attack Group Y 
and vice versa:

Group X = 24 x 24 x 24 x 24
Group Y = 24 x 24 x 24 x 24

So this is 24 possibility to the power of 8.

Final computation complexity at the very minimum is (24 to the power of 8) 
multiplied by roughly 4 attacks perhaps even 5 or 6 to finally destroy a 
group of numbers.

24 to the power of 8 = 110.075.314.176

I have already written a computer program which can solve this, however the 
computer program estimates it takes 19 hours on a 2.0 gigahertz AMD Athlon 
X2 core.

Using dual core could solve the problem over night, though I do not feel 
comfortable running my PC at night unattended.

So now I have the idea to make this program run when my computer is idling 
during the day, it should also be able to store it's progress so that it can 
continue after it was shutdown.

(Idea for now is to make it multi threaded and assign a low thread priority 
so it can run during the day when I use my computer and it's not doing much 
so it can use the reserve computational horse power).
(I still have to try these "idle/reverse" ideas to see which one works best 
without interrupting my web browsing or music listening too much ;))

My system has 4 GB of ram, so other ideas could be to store a data structure 
partially which could keep some computations so that it doesn't have to be 
done again... Though memory lookups might be a bit slow so not sure if that 
makes any sense.

I might also try GPU/Cuda since there seems to be lots of loops/reoccurences 
of the same computations that will happen over and over again... So maybe 
cuda can detect "same branch execution" and some "computations" and might 
speed it up, not sure about that.

Just the 8 index loops already cost a lot of instructions. Since there are 
only 24 permutation it would be enough to store it in 5 bits. Perhaps a 
rounded floating point which increases by 1/24 might be enough to trigger 
the 4th bit from incrementing when it actually needs to.
2x2x2x2x2 = 32  (it should increment bit 6 when the 5 bits reach 24).
So the idea here was to create 8 indexes from just 1 index being incremented 
to create the 8 combinations of indexes "instruction cheaply".
Not sure if this will work, just an idea I might try :)

Then those bits would still need to be extract and makes. So perhaps on 
older systems this is not efficient.

The 8 indexes need at least 3 instructions, 1 index increment, 1 
comparision, 1 jump.

The inner loop contains some while loops to increment attack index per 
number.

Each number has a "alive" variable which starts at 1.0 and is decreased 
everytime it's attacked.

Once a number is dead below 0.0000001 it's considered dead and can no longer 
attack.

(Since victory table was described as integers above this can also be read 
as: Alive starts at 100 and once it goes zero or negative it's dead).

Anyway the main reason why I wrote to this/these groups is that the numbers 
themselfes are not that larger and can fit in a byte, even a few bits.

Thus they will fit into SSE registers and such and I also know SSE has some 
permutations instructions.

I am not expert at SSE but I am wondering if there are perhaps some 
SSE1/2/3/4/5 instructions which can help at solving/computing this problem ?

Now the question you might be wondering about is ? What am I actually trying 
to compute ?

Well the final output could simply be the number of victories that each 
attack order has had per number. I am particullary interested in victories 
of number 4. So that would be sufficient for now.

Number 4 has 24 permutations.

So I want to know how each permutation of number 4 performs under all other 
circumstances/permutations of all other numbers/combinations and so forth.

This basically requires the entire set to be computed and then record number 
of victories for for example Group X number 4.

Only the results of one group needs to be outputted since the two groups are 
a mirror of each other.

Also during the development of the computer program I finally had a 
successfull implementation by keeping it as simple as possible and working 
with direct variables, instead of much more complex arrays.

Later on I made a more code efficient version which uses arrays/lookups and 
such. It was much harder to try the array approach at first since it becomes 
mindly complex and can obscure "vision to a solution".

So I suggest anybody trying to implement a solver for this to keep the code 
as simple as possible at first, since this is already an 8 dimensional 
problem with a 9th dimension.

Also it was interesting to see the Delphi for loops not allowing array 
fields as loop variables, so those loops will need to be fleshed out anyway, 
or one big linear loop could be used and then 8 indexes calculated from 
that.

That approach will probably be necessary for a cuda solution anyway... 32 
bit might be sufficient if remainders are reincluded in the conversion from 
thread/block dimensions to linear dimension back to 8 dimensional indexes.

Perhaps such computation might even be a bit quicker than the 3 instructions 
per loop.

For example 8 divisions and 8 remainders will be necessary to compute 8 
indexes from one big linear indexes. Since div/mod can be the same 
instruction this might only require 8 instructions to compute these loop 
indexes from a big linear index.
However computing the big linear index already requires between 6 or 10 
instructions.

So just to compute those 100+ billion data entries requires something like 
20 instructions just to be able to compute those loop indexes.

This is my finding with bigger data sets which consists out of small little 
data units... The closer the number of entries/data items get to the 
actually processing computing power the more time/processing/instructions 
are wasted on just computing these loop indexes.

It makes sense from this perspective: A problem of 100 items only needs 100 
loops. Thus lots of processing power remains for the data. But for 2 billion 
data items, the number of items already matches the gigaherts of the core... 
so the core is already swamped... with just loop index calculations.

I hope this reasoning/explanation convinces some CPU/GPU designers to 
include more instructions to compute loop indexes in all kinds of ways that 
could speed that up somewhat and safe some processing time.

Anyway... let me know if you think SSE could be of some help in solving this 
permutation/combination problem ?!

Also for the python newsgroup:

Maybe python is more powerfull than I thought and it has some weird/kinda crazy/cool new permutation/combination algorithmic classes that can solve these kinds of problems faster than just the average run of the mill general code ! ;)
(Or perhaps build in solver support, that will be my next try/posting :P :))

Bye,
  Skybuck. 


From marko at pacujo.net  Fri Dec  9 07:34:16 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 14:34:16 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <kp9b4chrk2qqsecasqlvbsk7rf57a16igt@4ax.com>
 <5848a4c7$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o2ac62$9vk$1@dont-email.me>
 <5848d64e$0$2901$c3e8da3$76491128@news.astraweb.com>
 <o2cbd8$5b2$1@dont-email.me>
 <CAPTjJmqABmbfUgiPGRdUFNae9Cfszftfr0dwJhHRBrsXUjTymw@mail.gmail.com>
 <mailman.176.1481225770.2253.python-list@python.org>
 <o2ck1k$51c$1@dont-email.me>
 <CAPTjJmo2kBbtwNpTa_r7=UynUz6CH3Z1i9BsYM7b29N=31gFxg@mail.gmail.com>
 <mailman.182.1481236319.2253.python-list@python.org>
 <o2cpof$r89$1@dont-email.me>
 <CAPTjJmpVOCA2KZ_qQ5wVWJsnM1-eTR91Vkktq3=c-=fbMwnSGA@mail.gmail.com>
 <mailman.188.1481244947.2253.python-list@python.org>
 <o2d1ku$gfo$1@dont-email.me>
 <CAPTjJmqmdXRPwRJh26FvV_=COauk9Am7LVA0_+DjQy48Zy8XdQ@mail.gmail.com>
 <mailman.192.1481251948.2253.python-list@python.org>
 <584a67c2$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CACL+1atSf8_UstdhdOeaRQ6bj4EkvZtCh_fQCLfNFBL3BYCZaA@mail.gmail.com>
 <mailman.196.1481281364.2253.python-list@python.org>
Message-ID: <8737hx1dtj.fsf@elektro.pacujo.net>

eryk sun <eryksun at gmail.com>:

> Windows NTFS doesn't normalize names to a canonical form. It also
> allows lone surrogate codes, which is invalid UTF-16.

Somewhat related, surrogate codes are invalid Unicode and shouldn't be
allowed in Unicode strings. However, Python does allow them.


Marko


From steve+python at pearwood.info  Fri Dec  9 10:26:15 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 10 Dec 2016 02:26:15 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me> <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
 <87bmwlodi6.fsf@elektro.pacujo.net>
 <584a6a9b$0$1621$c3e8da3$5496439d@news.astraweb.com>
 <877f791jcf.fsf@elektro.pacujo.net>
Message-ID: <584acd19$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Fri, 9 Dec 2016 09:34 pm, Marko Rauhamaa wrote:

> Steve D'Aprano <steve+python at pearwood.info>:
> 
>> On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:
>>> In Linux, "." and ".." are taboo.
>>
>> No that's incorrect. It isn't that . and .. are forbidden, but they are
>> reserved: every single directory in Unix file systems have a . and ..
>> directory entry. So they are legitimate directory names -- they're just
>> not names you can use for your own files, as they are already in use.
> 
> Same difference.

Of course they are different.

'\0' (the ASCII null character) is not a legal file name. You can't open a
file with that name, or (more relevant to this point) have a directory with
that name.

'..' and '.' are legal file names, or to be more specific, legal directory
names. You can have directories with those names, and they can appear in
paths.

py> os.path.isdir('..')
True
py> os.path.isdir('\0')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.3/genericpath.py", line 41, in isdir
    st = os.stat(s)
TypeError: embedded NUL character


See the difference?



>>> So is "".
>>
>> That's one way of looking at things... I'm not sure that the empty
>> string counts as a file name. Its more of the LACK of a file name.
>>
>> Besides, that would be ambiguous. Would "/home/steve/" mean my home
>> directory, or the file "" inside my home directory?
> 
> If Python had been the standard shell 

Python isn't a shell.


> when Unix came about, 

Ah, a counter-factual.


> you could  
> have defined pathnames as lists of strings. Then, everything would be
> unambigous and there wouldn't be any taboo or reserved names.

I really don't think so.

So what path does [] represent? How do you distinguish between the root
directory / and no pathname at all in a syntactically correct, unambiguous
manner, using only lists?

What path does [[]] represent? How about [[], []]?

Here's a nice one for you to think about:

L = []; L.append(L)

What path does L represent? How would it be represented in the file system?

What about this "path"? [None, 23, float('NAN'), {}]

What's the difference between ['a', 'b', 'c'] and ['a', ['b', ['c']]] as
path names? If there's no difference, are you concerned at the redundancy?
Is there a canonical form for paths?

How do you represent the current and previous directory in a list without
reserving identifiers for them?

How do you distinguish between '/home/steve' and '/home/steve/' for those
programs that (rightly or wrongly) need to distinguish them?

How do you handle alternate data streams? Or versioning information, for
systems like VMS that track that?

What about file systems where you can have multiple roots? E.g. Windows, C
\file versus D:\file and classic Mac Floppy:file and HardDrive:file?


Do you really imagine that if Python were a shell, and required system
administrators to write things like:

mount ['dev', 'dvd'] ['mount', 'dvd']

instead of

mount /dev/dvd /mount/dvd

that it would have been a success?

Hint: https://www.jwz.org/doc/worse-is-better.html



> BTW, guile allows *any* characters in an identifier. Even the empty name
> is a valid identifier:
> 
>    > (define #{}# "hello")
>    > #{}#
>    $1 = "hello"
>    (symbol->string '#{}#)
>    $2 = ""
>    > (symbol->string '#{.}#)
>    $4 = "."


And are Guile programs improved by having the empty string as an identify?

When was the last time you were coding and you thought "This program would
be so much easier to understand and maintain if only I could name this the
empty string"?

Does it simplify the process of one programmer talking to another programmer
about an algorithm to be able to use \n\n\n\0\n\n\n as an identifier?

Sometimes more is less.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From marko at pacujo.net  Fri Dec  9 13:40:14 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 09 Dec 2016 20:40:14 +0200
Subject: python 2.7.12 on Linux behaving differently than on Windows
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
 <87bmwlodi6.fsf@elektro.pacujo.net>
 <584a6a9b$0$1621$c3e8da3$5496439d@news.astraweb.com>
 <877f791jcf.fsf@elektro.pacujo.net>
 <584acd19$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <8760mtndyp.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> On Fri, 9 Dec 2016 09:34 pm, Marko Rauhamaa wrote:
>> Steve D'Aprano <steve+python at pearwood.info>:
>>> No that's incorrect. It isn't that . and .. are forbidden, but they
>>> are reserved: every single directory in Unix file systems have a .
>>> and .. directory entry. So they are legitimate directory names --
>>> they're just not names you can use for your own files, as they are
>>> already in use.
>> 
>> Same difference.
>
> Of course they are different.

You are not allowed to name your next file "." or ".." or "1/8".

>> you could have defined pathnames as lists of strings. Then,
>> everything would be unambigous and there wouldn't be any taboo or
>> reserved names.
>
> I really don't think so.
>
> So what path does [] represent?

That's the root directory.

> How do you distinguish between the root directory / and no pathname at
> all in a syntactically correct, unambiguous manner, using only lists?

I don't know what you mean.

> What path does [[]] represent? How about [[], []]?

Those are not lists of strings (see above).

> Here's a nice one for you to think about:
>
> L = []; L.append(L)
>
> What path does L represent?

That's not a list of strings.

> How would it be represented in the file system?
>
> What about this "path"? [None, 23, float('NAN'), {}]

That's not a list of strings.

> What's the difference between ['a', 'b', 'c'] and ['a', ['b', ['c']]] as
> path names?

Only the first one is a legal pathname.

> How do you represent the current and previous directory in a list
> without reserving identifiers for them?

My idea covered absolute pathnames only.

> How do you distinguish between '/home/steve' and '/home/steve/' for those
> programs that (rightly or wrongly) need to distinguish them?

I wouldn't. That distinction would have to be made some other way.

> How do you handle alternate data streams? Or versioning information, for
> systems like VMS that track that?

I am only talking about pathnames.

> What about file systems where you can have multiple roots? E.g. Windows, C
> \file versus D:\file and classic Mac Floppy:file and HardDrive:file?

I would only have one root.

> Do you really imagine that if Python were a shell, and required system
> administrators to write things like:
>
> mount ['dev', 'dvd'] ['mount', 'dvd']
>
> instead of
>
> mount /dev/dvd /mount/dvd
>
> that it would have been a success?

Dunno. I've been burned by bash so often that solid ground is what I
yearn for most.

>> BTW, guile allows *any* characters in an identifier. Even the empty name
>> is a valid identifier:
>
> And are Guile programs improved by having the empty string as an
> identify?

Point is, it's not at all impossible to remove the limitations on what
can go in names, filenames or otherwise.

> When was the last time you were coding and you thought "This program
> would be so much easier to understand and maintain if only I could
> name this the empty string"?

Having to be on constant lookout for corner cases is the problem.


Marko


From rosuav at gmail.com  Fri Dec  9 13:49:00 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 10 Dec 2016 05:49:00 +1100
Subject: python 2.7.12 on Linux behaving differently than on Windows
In-Reply-To: <8760mtndyp.fsf@elektro.pacujo.net>
References: <o2687e$ft7$1@dont-email.me>
 <22403f9a-c44a-38d9-ef04-f9bc7babd909@gmail.com>
 <mailman.87.1481039103.2253.python-list@python.org>
 <o26pll$fpk$1@dont-email.me>
 <5847c8b0$0$2888$c3e8da3$76491128@news.astraweb.com>
 <o28t7e$2s9$1@dont-email.me>
 <ag4g4ctmq839mkncj9o06jdk0pqn78m1f5@4ax.com>
 <mailman.124.1481119111.2253.python-list@python.org>
 <o29au2$ip2$1@dont-email.me> <easccjFefl8U1@mid.individual.net>
 <1481211447.860856.812710809.38EC435C@webmail.messagingengine.com>
 <6m2k4c1p3r833vpacug167fnilf4n2ip0r@4ax.com>
 <1481261555.1464014.813451961.71260688@webmail.messagingengine.com>
 <mailman.194.1481261558.2253.python-list@python.org>
 <87bmwlodi6.fsf@elektro.pacujo.net>
 <584a6a9b$0$1621$c3e8da3$5496439d@news.astraweb.com>
 <877f791jcf.fsf@elektro.pacujo.net>
 <584acd19$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <8760mtndyp.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmoJSCYaTtY9d7gF2rN857e875o7SBvpQcfTuHgQ-+kHAw@mail.gmail.com>

On Sat, Dec 10, 2016 at 5:40 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> How do you represent the current and previous directory in a list
>> without reserving identifiers for them?
>
> My idea covered absolute pathnames only.

Well, you're going to need to cope with relative pathnames somehow. I
suppose you could do:

[RELATIVE, PARENT, PARENT, "subdir", "file"]

but I'd rather use "../../subdir/file", myself.

ChrisA


From jon+usenet at unequivocal.eu  Fri Dec  9 16:34:38 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Fri, 9 Dec 2016 21:34:38 -0000 (UTC)
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
Message-ID: <slrno4m8tv.298.jon+usenet@sable.unequivocal.eu>

On 2016-12-09, DFS <nospam at dfs.com> wrote:
> import sys as y,nntplib as t,datetime as d
> s='<news server>'
> g=y.argv[1]
> n=t.NNTP(s,119,'<usr>','<pw>')
> r,a,b,e,gn=n.group(g)
> def printStat(st,hd,rg):
> 	r,d=n.xhdr(st,'%s-%s'%rg)
> 	p=[]
> 	for i in range(len(d)):
> 		v=d[i][1]
> 		if st=='Subject':v=v[4:] if v[:3]=='Re:' else v
> 		p.append(v)
> 	x=[(i,p.count(i)) for i in set(p)]
> 	x.sort(key=lambda s:(-s[1],s[0].lower()))
> 	print('Posts  %s %s'%(len(set(p)),hd))
> 	for v in x: print(' %s     %s'%(v[1],v[0]))
> 	print
> print 'As of '+d.datetime.now().strftime("%I:%M%p %B %d, %Y") + '\n'
> m=(int(e)-int(y.argv[3])+1,int(e))
> printStat("From","Posters",m)
> printStat("Subject","Subjects",m)
> printStat("User-Agent","User-Agents",m)
> n.quit()

Was there ever an "International Obfuscated Python Code Contest"? ;-)


From rosuav at gmail.com  Fri Dec  9 16:48:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 10 Dec 2016 08:48:56 +1100
Subject: CLP stats: last 500 posts
In-Reply-To: <slrno4m8tv.298.jon+usenet@sable.unequivocal.eu>
References: <o2f6bn$70u$2@dont-email.me>
 <slrno4m8tv.298.jon+usenet@sable.unequivocal.eu>
Message-ID: <CAPTjJmoC+KVO6-HWofuZWDW-kaCRd4KthfGEP12RxxCHEZOTdg@mail.gmail.com>

On Sat, Dec 10, 2016 at 8:34 AM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
>
> Was there ever an "International Obfuscated Python Code Contest"? ;-)

I don't know, but if so, here's my entry:

print(*([0,"Fizz","Buzz","Fizzbuzz"][[3,0,0,1,0,2,1,0,0,1,2,0,1,0,0][i%15]]or
i for i in range(1,51)))

ChrisA


From space.ship.traveller at gmail.com  Fri Dec  9 18:11:32 2016
From: space.ship.traveller at gmail.com (space.ship.traveller at gmail.com)
Date: Fri, 9 Dec 2016 15:11:32 -0800 (PST)
Subject: Running python from pty without prompt
Message-ID: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>

Hello.

I'm working on a script runner for Atom.

https://github.com/ioquatix/script-runner

We are trying to understand how to make python work well. I'll use a comparison to the ruby executable because it's convenient to explain the problem.

When you invoke `ruby` from a pty, you get no output (as opposed to `irb`, interactive ruby [shell]). You can write a script to stdin, and send Ctrl-D (EOT / 0x04). Then, ruby will execute the script. stdin is not closed so programs that expect interactive input will work correctly.

When we run python in the same way, we get an output prompt. As soon as a function like `input` is called, the program is interrupted.

I'm happy to hear feedback about how this should work. Perhaps our expectations are wrong, or what we are doing is wrong.

One option which we've been considering is saving the input as a file and executing that. But, it's not as clean compared to simply writing a string to stdin of a running interpreter and then 0x04.

Thanks for any and all help.

Kind regards,
Samuel


From torriem at gmail.com  Fri Dec  9 19:45:29 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 9 Dec 2016 17:45:29 -0700
Subject: Running python from pty without prompt
In-Reply-To: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <026a77ab-e0db-13e7-9095-271a5427164b@gmail.com>

On 12/09/2016 04:11 PM, space.ship.traveller at gmail.com wrote:
> Hello.
> 
> I'm working on a script runner for Atom.
> 
> https://github.com/ioquatix/script-runner
> 
> We are trying to understand how to make python work well. I'll use a
> comparison to the ruby executable because it's convenient to explain
> the problem.
> 
> When you invoke `ruby` from a pty, you get no output (as opposed to
> `irb`, interactive ruby [shell]). You can write a script to stdin,
> and send Ctrl-D (EOT / 0x04). Then, ruby will execute the script.
> stdin is not closed so programs that expect interactive input will
> work correctly.
> 
> When we run python in the same way, we get an output prompt. As soon
> as a function like `input` is called, the program is interrupted.
> 
> I'm happy to hear feedback about how this should work. Perhaps our
> expectations are wrong, or what we are doing is wrong.

Not sure I understand the issue here.  You can pipe a script to Python
and it runs it without any immediate-mode prompt.  I think python only
shows the REPL prompt if you are attached to a pty.

But if the script is piped into Python or ruby, I don't know how we
could expect raw_input() or input() to function.

> One option which we've been considering is saving the input as a file
> and executing that. But, it's not as clean compared to simply writing
> a string to stdin of a running interpreter and then 0x04.

An intermediate file is unnecessary as you can pipe a script into Python.


From torriem at gmail.com  Fri Dec  9 19:47:32 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 9 Dec 2016 17:47:32 -0700
Subject: Running python from pty without prompt
In-Reply-To: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <c7afd118-10df-2121-3327-fff9705555f0@gmail.com>

On 12/09/2016 04:11 PM, space.ship.traveller at gmail.com wrote:
> When you invoke `ruby` from a pty, you get no output (as opposed to
> `irb`, interactive ruby [shell]). You can write a script to stdin,
> and send Ctrl-D (EOT / 0x04). Then, ruby will execute the script.
> stdin is not closed so programs that expect interactive input will
> work correctly.
> 
> When we run python in the same way, we get an output prompt. As soon
> as a function like `input` is called, the program is interrupted.

Nevermind my other post. I understand what you are saying.


From space.ship.traveller at gmail.com  Fri Dec  9 20:27:51 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Fri, 9 Dec 2016 17:27:51 -0800 (PST)
Subject: Running python from pty without prompt
In-Reply-To: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <00dd931a-48c0-400e-acc3-455259403b24@googlegroups.com>

Just in case it's not clear, this is running on a (virtual) PTY.


From steve+python at pearwood.info  Fri Dec  9 20:39:53 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 10 Dec 2016 12:39:53 +1100
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
Message-ID: <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>

On Sat, 10 Dec 2016 08:07 am, DFS wrote:

> 
> As of 04:04PM December 09, 2016
> 
> Posts  85 Posters
[...]


Interesting stats, but couldn't you have post-processed the results to avoid
including the defamatory spam posts?

Your post is likely to be removed from the official web archive as it
contains defamatory material.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Fri Dec  9 20:43:22 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 10 Dec 2016 12:43:22 +1100
Subject: Running python from pty without prompt
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <584b5dbc$0$1591$c3e8da3$5496439d@news.astraweb.com>

On Sat, 10 Dec 2016 10:11 am, space.ship.traveller at gmail.com wrote:

> Hello.
> 
> I'm working on a script runner for Atom.
> 
> https://github.com/ioquatix/script-runner
> 
> We are trying to understand how to make python work well. I'll use a
> comparison to the ruby executable because it's convenient to explain the
> problem.

Could you show a small, simple demonstration of both the Ruby code that
works they way you want, and the Python code that behaves differently?

Preferably one that runs straight from vanilla Python without any
third-party libraries, including your script-runner.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From torriem at gmail.com  Fri Dec  9 21:16:56 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 9 Dec 2016 19:16:56 -0700
Subject: Running python from pty without prompt
In-Reply-To: <584b5dbc$0$1591$c3e8da3$5496439d@news.astraweb.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <584b5dbc$0$1591$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <1738a9cc-a9d8-f096-92c0-333981208a40@gmail.com>

On 12/09/2016 06:43 PM, Steve D'Aprano wrote:
> On Sat, 10 Dec 2016 10:11 am, space.ship.traveller at gmail.com wrote:
> 
>> Hello.
>>
>> I'm working on a script runner for Atom.
>>
>> https://github.com/ioquatix/script-runner
>>
>> We are trying to understand how to make python work well. I'll use a
>> comparison to the ruby executable because it's convenient to explain the
>> problem.
> 
> Could you show a small, simple demonstration of both the Ruby code that
> works they way you want, and the Python code that behaves differently?
> 
> Preferably one that runs straight from vanilla Python without any
> third-party libraries, including your script-runner.

I imagine they want to feed Python a combination of a script and also
standard-in input in the same stream. Something like:

$ python << EOF
a = input("Enter your name: ")
print a
^D
bob
EOF

Where ^D is a literal control character (ctrl-v, control-d on the
terminal, but does not actually close the stream or signify the end).
This would be piped into Python's standard-in where it would feed python
both the script to run, and also input to feed the script.  Apparently
ruby can do this.

Did I understand this correctly, space.ship.traveller?



From clvanwall at gmail.com  Fri Dec  9 22:33:20 2016
From: clvanwall at gmail.com (clvanwall)
Date: Fri, 09 Dec 2016 21:33:20 -0600
Subject: Is there a replement for bsddb in python3?
Message-ID: <lvoloqg7dm5t99vjabhy80m7.1481340684122@email.android.com>




I have been looking for a simple indexed file database and Berkley-db was what I waa used to. ?I found that bsddb module was removed from Python3. ?Is there a replacement for it?
John ?Van Walleghen?Sent from my Galaxy Tab? A

From no.email at nospam.invalid  Fri Dec  9 23:35:52 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Fri, 09 Dec 2016 20:35:52 -0800
Subject: Is there a replement for bsddb in python3?
References: <lvoloqg7dm5t99vjabhy80m7.1481340684122@email.android.com>
 <mailman.204.1481340804.2253.python-list@python.org>
Message-ID: <87wpf82yfr.fsf@nightsong.com>

clvanwall <clvanwall at gmail.com> writes:
> I found that bsddb module was removed from Python3.  Is there a
> replacement for it?

Try "dbm" which has a few options for the underlying engine.
Alternatively, try sqlite3.


From ben+python at benfinney.id.au  Fri Dec  9 23:58:16 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 10 Dec 2016 15:58:16 +1100
Subject: Is there a replement for bsddb in python3?
References: <lvoloqg7dm5t99vjabhy80m7.1481340684122@email.android.com>
Message-ID: <85zik41itz.fsf@benfinney.id.au>

clvanwall <clvanwall at gmail.com> writes:

> I found that bsddb module was removed from Python3. ?Is there a
> replacement for it?

The ?dbm? library is what you need now, I believe
<URL:https://docs.python.org/3/library/dbm.html>.

    dbm is a generic interface to variants of the DBM database [?]
    There is a third party interface to the Oracle Berkeley DB.

-- 
 \       ?Give a man a fish, and you'll feed him for a day; give him a |
  `\    religion, and he'll starve to death while praying for a fish.? |
_o__)                                                       ?Anonymous |
Ben Finney



From tjreedy at udel.edu  Sat Dec 10 00:36:42 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 10 Dec 2016 00:36:42 -0500
Subject: CLP stats: last 500 posts
In-Reply-To: <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2g497$3vs$1@blaine.gmane.org>

On 12/9/2016 8:39 PM, Steve D'Aprano wrote:
> On Sat, 10 Dec 2016 08:07 am, DFS wrote:
>
>>
>> As of 04:04PM December 09, 2016
>>
>> Posts  85 Posters
> [...]
>
>
> Interesting stats, but couldn't you have post-processed the results to avoid
> including the defamatory spam posts?
>
> Your post is likely to be removed from the official web archive as it
> contains defamatory material.

Reading the news.gmane.org mirror, I never received it.

-- 
Terry Jan Reedy



From vek.m1234 at gmail.com  Sat Dec 10 02:06:23 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sat, 10 Dec 2016 12:36:23 +0530
Subject: % string formatting - what special method is used for %d?
Message-ID: <o2g9f1$3oi$1@dont-email.me>

When we do:

print '%s %d' % ('hello', 10)

what special method is being invoked internally within the string-
format-specifier? 

format() invokes format__
print invokes __str__

I'm basically trying to make sense of:

raise TypeError('urkle urkle %s' % list(dictionary))
<=> raise TypeError('urkle urkle %s' % [ key1, val1, key2, val2 ]

So, the % operator reads the format specifier and notices %s and 
therefore calls __str__ in the list class to figure out how to represent 
[ key1, val1, key2, val2 ].

However what if I use %d? How do the other format specs work?


From steve+python at pearwood.info  Sat Dec 10 03:13:15 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 10 Dec 2016 19:13:15 +1100
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <o2fvfb$cpd$1@dont-email.me>
Message-ID: <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>

On Sat, 10 Dec 2016 03:15 pm, DFS wrote:

> On 12/09/2016 08:39 PM, Steve D'Aprano wrote:
>> On Sat, 10 Dec 2016 08:07 am, DFS wrote:
>>
>>>
>>> As of 04:04PM December 09, 2016
>>>
>>> Posts  85 Posters
>> [...]
>>
>>
>> Interesting stats, but couldn't you have post-processed the results
>> to avoid including the defamatory spam posts?
> 
> 
> Normally I don't censor, at all.  But the spams are apparently way
> off-topic, so I'll filter out Subjects containing certain keywords.

Its not just the Subject, but also the fake Sender. There are at least five
distinct senders which are (apparently) defamatory messages in Italian.
They're all pretty obvious spam, in all caps, with various email addresses.

"... MEGLIO ..." <uccipaducci at gmx dot com>



> The spammer will still be counted, but the stats won't show all those
> stupid Subjects.

I don't mind if the spammer is counted. They probably should be collated
together, and count as a single sender using multiple addresses. But the
false name should be expunged or elided.


>> Your post is likely to be removed from the official web archive as
>> it contains defamatory material.
> 
> Google seems to archive most Usenet posts, but there is no 'official web
> archive'.  Nor will those posts be auto-removed from GoogleGroups for
> their content.

comp.lang.python is a mirror of the python-list at python dot org mailing
list, which has an official web archive:

https://mail.python.org/pipermail/python-list/

There are many unofficial ones as well.

There are a few other people who are banned from the mailing list but still
post to the newsgroup.


> Why does this wackjob post all that Italian-language spam to clp anyway?

Why do wackjobs do anything? He has a bee in his bonnet about some other
fellow, I don't even know if its a politician or just some guy he knows,
and (apparently) spams dozens of newsgroups with defamatory posts accusing
him of being a paedophile, a criminal, and more.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Dec 10 03:52:02 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 10 Dec 2016 19:52:02 +1100
Subject: % string formatting - what special method is used for %d?
References: <o2g9f1$3oi$1@dont-email.me>
Message-ID: <584bc233$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Sat, 10 Dec 2016 06:06 pm, Veek M wrote:

> When we do:
> 
> print '%s %d' % ('hello', 10)
> 
> what special method is being invoked internally within the string-
> format-specifier?

%d requires the argument to be an int, or able to be converted to int using
the __int__ special method.


py> class X(object):
...     def __int__(self):
...             return 42
...
py> "%d" % X()
'42'



> format() invokes format__
> print invokes __str__

print actually invokes __str__ or __repr__, whichever is available.



> I'm basically trying to make sense of:
> 
> raise TypeError('urkle urkle %s' % list(dictionary))
> <=> raise TypeError('urkle urkle %s' % [ key1, val1, key2, val2 ]


The raise TypeError part of the code is irrelevant to your question. You
should always simplify your code to only the part that is relevant. 

raise TypeError(some_string)

behaves the same regardless of how some_string is made.


> So, the % operator reads the format specifier and notices %s and
> therefore calls __str__ in the list class to figure out how to represent
> [ key1, val1, key2, val2 ].
> 
> However what if I use %d? How do the other format specs work?


The format specifiers are similar to these:

%s => str(obj), which ends up calling __str__ or __repr__

%r => repr(obj), which ends up calling __repr__ or __str__

%c => chr(obj), or obj must be a string of length 1

%d %i %u => int(obj), which ends up calling __int__

%x %X => int(obj), then convert to hexadecimal

%o => int(obj), then convert to octal

%e %E %f %g %G => float(obj), which ends up calling __float__

%% => a literal % sign





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From tjreedy at udel.edu  Sat Dec 10 05:28:47 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 10 Dec 2016 05:28:47 -0500
Subject: CLP stats: last 500 posts
In-Reply-To: <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <o2fvfb$cpd$1@dont-email.me>
 <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2glcs$72s$1@blaine.gmane.org>

On 12/10/2016 3:13 AM, Steve D'Aprano wrote:
> On Sat, 10 Dec 2016 03:15 pm, DFS wrote:

>> Normally I don't censor, at all.  But the spams are apparently way
>> off-topic, so I'll filter out Subjects containing certain keywords.

python-list is a spam-moderated list. 95+% of spam is filtered out.

> Its not just the Subject, but also the fake Sender. There are at least five
> distinct senders which are (apparently) defamatory messages in Italian.

This person actively evades our filters.

> They're all pretty obvious spam, in all caps, with various email addresses.
>
> "... MEGLIO ..." <uccipaducci at gmx dot com>
>
>
>
>> The spammer will still be counted,

Why reward someone who actively evades defenses?  If you want to count 
spam, it is mostly missing, at least as far as python-list is concerned.

> comp.lang.python is a mirror of the python-list at python dot org mailing
> list, which has an official web archive:
>
> https://mail.python.org/pipermail/python-list/

These slanderous posts, in particular, are hand-removed from the archive 
when they get past the automatic filters.  They are no more part of 
python-list than other spam.

-- 
Terry Jan Reedy




From iranna.gani28 at gmail.com  Sat Dec 10 05:33:54 2016
From: iranna.gani28 at gmail.com (Iranna Mathapati)
Date: Sat, 10 Dec 2016 16:03:54 +0530
Subject: Pexpect
Message-ID: <CAA4=HD6CuGr4EkKzVGxvAu7bJBBNmz_xsB7qwsyJ5Ge3Hm3L-Q@mail.gmail.com>

Hi Team,



The fallowing script(function) is working if its in PDB mode(trace line by
line) but same script is not working if its remove/comment PDB and run one
shot.



log.info("Time out expection")

            uut_con1 = pexpect.spawn('telnet %s'%power_ip)

            uut_con1.logfile = sys.stdout

            uut_con1.delaybeforesend = None

            time.sleep(30)

            uut_con1.send("%s\r" % username)

            uut_con1.expect([r"Password:"],timeout=30)

            uut_con1.send("%s\r" % pass_word)

            time.sleep(30)

            uut_con1.send("\r")

            uut_con1.expect([r'#?'],timeout=30)

            uut_con1.send("power outlets %d off\r"%outlet)

            uut_con1.expect([r'wish to turn outlet'],timeout=30)

            uut_con1.send("y\r")

            uut_con1.send("\r")

            uut_con1.expect([r'#?'],timeout=30)

            uut_con1.send("power outlets %d on\r"%outlet)

            uut_con1.expect([r'wish to turn outlet'],timeout=30)

            uut_con1.send("y\r")

            uut_con1.send("\r")

            uut_con1.expect([r'#?'],timeout=30)

            uut_con1.close()

            uut_con.close()

            time.sleep(300)


check_prompt(self,username,pass_word,dev_ip,dev_port,power_ip,outlet)



*I got fallowing error:*



2016-12-10T02:24:42: %aetest-INFO: Time out expection

Trying 172.31.206.143...

Connected to 172.31.206.143.

Escape character is '^]'.

Login for PX2 CLI

Username: admin

power outlets 21 off



Welcome to PX2 CLI!

Last login: 2001-06-30 11:53:59 EDT [CLI (Telnet) from 172.31.144.6]



#

# power outlets 21 off



yower outlets 21 on outlet 21 off? [y/n] y



power outlets 21 on

# 2016-12-10T02:26:12: %aetest-ERROR: Caught exception during execution:

2016-12-10T02:26:13: %aetest-ERROR: Traceback (most recent call last):

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/rajtamil/pyATS2.7/hlite/eor/systest/scripts/NAT/ATS_CLEAN_N9K.py",
line 354, in connect_devices

2016-12-10T02:26:13: %aetest-ERROR:
check_prompt(self,username,pass_word,dev_ip,dev_port,power_ip,outlet)

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/rajtamil/pyATS2.7/hlite/eor/systest/scripts/NAT/ATS_CLEAN_N9K.py",
line 282, in check_prompt

2016-12-10T02:26:13: %aetest-ERROR:     uut_con1.expect([r'wish to turn
outlet'],timeout=30)

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/svanalin/pyats2/lib/python2.7/site-packages/pexpect/spawnbase.py",
line 321, in expect

2016-12-10T02:26:13: %aetest-ERROR:     timeout, searchwindowsize, async)

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/svanalin/pyats2/lib/python2.7/site-packages/pexpect/spawnbase.py",
line 345, in expect_list

2016-12-10T02:26:13: %aetest-ERROR:     return exp.expect_loop(timeout)

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/svanalin/pyats2/lib/python2.7/site-packages/pexpect/expect.py",
line 107, in expect_loop

2016-12-10T02:26:13: %aetest-ERROR:     return self.timeout(e)

2016-12-10T02:26:13: %aetest-ERROR:   File
"/auto/n3k-qa/CODC/svanalin/pyats2/lib/python2.7/site-packages/pexpect/expect.py",
line 70, in timeout

2016-12-10T02:26:13: %aetest-ERROR:     raise TIMEOUT(msg)

2016-12-10T02:26:13: %aetest-ERROR: TIMEOUT: Timeout exceeded.

2016-12-10T02:26:13: %aetest-ERROR: <pexpect.pty_spawn.spawn object at
0xf686d5ec>

2016-12-10T02:26:13: %aetest-ERROR: command: /usr/bin/telnet

2016-12-10T02:26:13: %aetest-ERROR: args: ['/usr/bin/telnet',
'172.31.206.143']

2016-12-10T02:26:13: %aetest-ERROR: buffer (last 100 chars): ' 21 off?
[y/n] y\r\n\r\npower outlets 21 on\r\n# '

2016-12-10T02:26:13: %aetest-ERROR: before (last 100 chars): ' 21 off?
[y/n] y\r\n\r\npower outlets 21 on\r\n# '

2016-12-10T02:26:13: %aetest-ERROR: after: <class
'pexpect.exceptions.TIMEOUT'>

2016-12-10T02:26:13: %aetest-ERROR: match: None

2016-12-10T02:26:13: %aetest-ERROR: match_index: None

2016-12-10T02:26:13: %aetest-ERROR: exitstatus: None

2016-12-10T02:26:13: %aetest-ERROR: flag_eof: False

2016-12-10T02:26:13: %aetest-ERROR: pid: 9163

2016-12-10T02:26:13: %aetest-ERROR: child_fd: 18

2016-12-10T02:26:13: %aetest-ERROR: closed: False

2016-12-10T02:26:13: %aetest-ERROR: timeout: 30

2016-12-10T02:26:13: %aetest-ERROR: delimiter: <class
'pexpect.exceptions.EOF'>

2016-12-10T02:26:13: %aetest-ERROR: logfile: <open file '<stdout>', mode
'w' at 0xf7712078>

2016-12-10T02:26:13: %aetest-ERROR: logfile_read: None

2016-12-10T02:26:13: %aetest-ERROR: logfile_send: None

2016-12-10T02:26:13: %aetest-ERROR: maxread: 2000

2016-12-10T02:26:13: %aetest-ERROR: ignorecase: False

2016-12-10T02:26:13: %aetest-ERROR: searchwindowsize: None

2016-12-10T02:26:13: %aetest-ERROR: delaybeforesend: None

2016-12-10T02:26:13: %aetest-ERROR: delayafterclose: 0.1

2016-12-10T02:26:13: %aetest-ERROR: delayafterterminate: 0.1

2016-12-10T02:26:13: %aetest-ERROR: searcher: searcher_re:

2016-12-10T02:26:13: %aetest-ERROR:     0: re.compile("wish to turn outlet")

2016-12-10T02:26:13: %aetest-INFO: The result of subsection connect_devices
is => ERRORED





Regards,
Iranna M


From bisy2648 at gmail.com  Sat Dec 10 06:48:27 2016
From: bisy2648 at gmail.com (busybizz bizz)
Date: Sat, 10 Dec 2016 03:48:27 -0800 (PST)
Subject: Painting Contractors in Vasant nagar | Painting Contractors in Rajaji
 nagar
Message-ID: <f0f8f70a-5382-4559-922b-82afd703ee28@googlegroups.com>

Painting Contractors in Bangalore - List of top painting services, works, companies in Bangalore and get painting price, companies contact addresses, phone numbers, ratings and evaluations right away on busybizz.com
http://busybizz.com/Painting-Contractors-Bangalore.php


From steve+python at pearwood.info  Sat Dec 10 09:43:21 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 11 Dec 2016 01:43:21 +1100
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <o2fvfb$cpd$1@dont-email.me>
 <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>
 <o2glcs$72s$1@blaine.gmane.org>
 <mailman.207.1481365742.2253.python-list@python.org>
Message-ID: <584c148a$0$1583$c3e8da3$5496439d@news.astraweb.com>

On Sat, 10 Dec 2016 09:28 pm, Terry Reedy wrote:

>>> The spammer will still be counted,
> 
> Why reward someone who actively evades defenses?  If you want to count
> spam, it is mostly missing, at least as far as python-list is concerned.

Its not a reward. Spammers are not like trolls, they don't hang around to
see the result of their posts. There no evidence at all that this Italian
spammer is looking for replies or responses to his(?) posts. He apparently
just fires them out.

I think that it is relevant that comp.lang.python receives X spam messages
from a certain person. It gives a picture of the health of the newsgroup:
how much of it is spam? Hopefully only a small amount.


> These slanderous posts, in particular, are hand-removed from the archive
> when they get past the automatic filters.  They are no more part of
> python-list than other spam.

Indeed. But although c.l.p is a mirror of the mailing list, it is not a
*perfect* mirror. The two do diverge: some things go to the mailing list
but apparently never make it to the newsgroup, and some things get to the
newsgroup but don't make it to the mailing list.

The stats generated by DFS are relevant to that.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From skip.montanaro at gmail.com  Sat Dec 10 11:15:15 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sat, 10 Dec 2016 10:15:15 -0600
Subject: CLP stats: last 500 posts
In-Reply-To: <o2glcs$72s$1@blaine.gmane.org>
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <o2fvfb$cpd$1@dont-email.me>
 <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>
 <o2glcs$72s$1@blaine.gmane.org>
Message-ID: <CANc-5UxMYuzj30-KD76US4JFuH1gs_bxagTO71AZJ70p1FadjQ@mail.gmail.com>

On Sat, Dec 10, 2016 at 4:28 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> comp.lang.python is a mirror of the python-list at python dot org mailing
>> list, which has an official web archive:
>>
>> https://mail.python.org/pipermail/python-list/
>
>
> These slanderous posts, in particular, are hand-removed from the archive
> when they get past the automatic filters.  They are no more part of
> python-list than other spam.

Are they still getting past SpamBayes? A couple months ago, I trained
the instance on m.p.o on a few of those spams. I haven't seen any
others since then, and no messages on the postmaster list discussing
them. I just skimmed the archives for November and December but saw no
examples. Generally, when Ralf or Mark delete spams, then manually
rewrite the Subject: and From: headers and zero out the message body.
I saw nothing like those placeholders either.

Skip


From best_lay at yahoo.com  Sat Dec 10 12:06:50 2016
From: best_lay at yahoo.com (Wildman)
Date: Sat, 10 Dec 2016 11:06:50 -0600
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
Message-ID: <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>

On Fri, 09 Dec 2016 16:07:16 -0500, DFS wrote:

> code (py2.7)
> --------------------------------------------------------------
> import sys as y,nntplib as t,datetime as d
> s='<news server>'
> g=y.argv[1]
> n=t.NNTP(s,119,'<usr>','<pw>')
> r,a,b,e,gn=n.group(g)
> def printStat(st,hd,rg):
> 	r,d=n.xhdr(st,'%s-%s'%rg)
> 	p=[]
> 	for i in range(len(d)):
> 		v=d[i][1]
> 		if st=='Subject':v=v[4:] if v[:3]=='Re:' else v
> 		p.append(v)
> 	x=[(i,p.count(i)) for i in set(p)]
> 	x.sort(key=lambda s:(-s[1],s[0].lower()))
> 	print('Posts  %s %s'%(len(set(p)),hd))
> 	for v in x: print(' %s     %s'%(v[1],v[0]))
> 	print
> print 'As of '+d.datetime.now().strftime("%I:%M%p %B %d, %Y") + '\n'
> m=(int(e)-int(y.argv[3])+1,int(e))
> printStat("From","Posters",m)
> printStat("Subject","Subjects",m)
> printStat("User-Agent","User-Agents",m)
> n.quit()
> --------------------------------------------------------------
> 
> usage on Windows:
> $ python stats.py group last N
> $ python stats.py comp.lang.python last 500

Do you happen to have a translation of the code that will
run on Linux?

$ ./nntp.py comp.lang.python last 500
Traceback (most recent call last):
  File "./nntp.py", line 7, in <module>
    n=t.NNTP(s,119,'<usr>','<pw>')
  File "/usr/lib/python2.7/nntplib.py", line 119, in __init__
    self.sock = socket.create_connection((host, port))
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Sat Dec 10 12:59:57 2016
From: best_lay at yahoo.com (Wildman)
Date: Sat, 10 Dec 2016 11:59:57 -0600
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>
 <o2he34$kbt$1@dont-email.me>
Message-ID: <B8SdnSgvJtUA39HFnZ2dnUU7-V2dnZ2d@giganews.com>

On Sat, 10 Dec 2016 12:31:33 -0500, DFS wrote:

> On 12/10/2016 12:06 PM, Wildman wrote:
>> On Fri, 09 Dec 2016 16:07:16 -0500, DFS wrote:
>>
>>> code (py2.7)
>>> --------------------------------------------------------------
>>> import sys as y,nntplib as t,datetime as d
>>> s='<news server>'
>>> g=y.argv[1]
>>> n=t.NNTP(s,119,'<usr>','<pw>')
>>> r,a,b,e,gn=n.group(g)
>>> def printStat(st,hd,rg):
>>> 	r,d=n.xhdr(st,'%s-%s'%rg)
>>> 	p=[]
>>> 	for i in range(len(d)):
>>> 		v=d[i][1]
>>> 		if st=='Subject':v=v[4:] if v[:3]=='Re:' else v
>>> 		p.append(v)
>>> 	x=[(i,p.count(i)) for i in set(p)]
>>> 	x.sort(key=lambda s:(-s[1],s[0].lower()))
>>> 	print('Posts  %s %s'%(len(set(p)),hd))
>>> 	for v in x: print(' %s     %s'%(v[1],v[0]))
>>> 	print
>>> print 'As of '+d.datetime.now().strftime("%I:%M%p %B %d, %Y") + '\n'
>>> m=(int(e)-int(y.argv[3])+1,int(e))
>>> printStat("From","Posters",m)
>>> printStat("Subject","Subjects",m)
>>> printStat("User-Agent","User-Agents",m)
>>> n.quit()
>>> --------------------------------------------------------------
>>>
>>> usage on Windows:
>>> $ python stats.py group last N
>>> $ python stats.py comp.lang.python last 500
>>
>> Do you happen to have a translation of the code that will
>> run on Linux?
>>
>> $ ./nntp.py comp.lang.python last 500
>> Traceback (most recent call last):
>>   File "./nntp.py", line 7, in <module>
>>     n=t.NNTP(s,119,'<usr>','<pw>')
>>   File "/usr/lib/python2.7/nntplib.py", line 119, in __init__
>>     self.sock = socket.create_connection((host, port))
>>   File "/usr/lib/python2.7/socket.py", line 553, in create_connection
>>     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
>> socket.gaierror: [Errno -2] Name or service not known
> 
> 
> That code runs unchanged on py2.7 on Linux (I just now tested it).
> 
> You just need to put in your own credentials for the newsserver, user 
> and password (lines 2 and 4).

OK, thanks.  That didn't occur to me although it should have.
 
-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From __peter__ at web.de  Sat Dec 10 13:06:47 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 10 Dec 2016 19:06:47 +0100
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>
Message-ID: <o2hg7j$qha$1@blaine.gmane.org>

Wildman via Python-list wrote:

> On Fri, 09 Dec 2016 16:07:16 -0500, DFS wrote:
> 
>> code (py2.7)
>> --------------------------------------------------------------
>> import sys as y,nntplib as t,datetime as d
>> s='<news server>'
>> g=y.argv[1]
>> n=t.NNTP(s,119,'<usr>','<pw>')
>> r,a,b,e,gn=n.group(g)
>> def printStat(st,hd,rg):
>> r,d=n.xhdr(st,'%s-%s'%rg)
>> p=[]
>> for i in range(len(d)):
>> v=d[i][1]
>> if st=='Subject':v=v[4:] if v[:3]=='Re:' else v
>> p.append(v)
>> x=[(i,p.count(i)) for i in set(p)]
>> x.sort(key=lambda s:(-s[1],s[0].lower()))
>> print('Posts  %s %s'%(len(set(p)),hd))
>> for v in x: print(' %s     %s'%(v[1],v[0]))
>> print
>> print 'As of '+d.datetime.now().strftime("%I:%M%p %B %d, %Y") + '\n'
>> m=(int(e)-int(y.argv[3])+1,int(e))
>> printStat("From","Posters",m)
>> printStat("Subject","Subjects",m)
>> printStat("User-Agent","User-Agents",m)
>> n.quit()
>> --------------------------------------------------------------
>> 
>> usage on Windows:
>> $ python stats.py group last N
>> $ python stats.py comp.lang.python last 500
> 
> Do you happen to have a translation of the code that will
> run on Linux?
> 
> $ ./nntp.py comp.lang.python last 500
> Traceback (most recent call last):
>   File "./nntp.py", line 7, in <module>
>     n=t.NNTP(s,119,'<usr>','<pw>')
>   File "/usr/lib/python2.7/nntplib.py", line 119, in __init__
>     self.sock = socket.create_connection((host, port))
>   File "/usr/lib/python2.7/socket.py", line 553, in create_connection
>     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
> socket.gaierror: [Errno -2] Name or service not known
> 

That's not a Linux problem. For the code to run in 

>> s='<news server>'

>> n=t.NNTP(s,119,'<usr>','<pw>')

you need to replace the '<...>' strings with a real news server, user, and 
password. If you use Gmane no password is required:

n = t.NNTP("news.gmane.org")

However, they use a different name for the "comp.lang.python" group, so you 
have to modify the command line accordingly:

$ python stats.py gmane.comp.python.general last 500




From tjreedy at udel.edu  Sat Dec 10 15:27:24 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 10 Dec 2016 15:27:24 -0500
Subject: CLP stats: last 500 posts
In-Reply-To: <584c148a$0$1583$c3e8da3$5496439d@news.astraweb.com>
References: <o2f6bn$70u$2@dont-email.me>
 <584b5cec$0$1591$c3e8da3$5496439d@news.astraweb.com>
 <o2fvfb$cpd$1@dont-email.me>
 <584bb91c$0$1584$c3e8da3$5496439d@news.astraweb.com>
 <o2glcs$72s$1@blaine.gmane.org>
 <mailman.207.1481365742.2253.python-list@python.org>
 <584c148a$0$1583$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2hof8$gt6$1@blaine.gmane.org>

On 12/10/2016 9:43 AM, Steve D'Aprano wrote:
> On Sat, 10 Dec 2016 09:28 pm, Terry Reedy wrote:
>
>>>> The spammer will still be counted,
>>
>> Why reward someone who actively evades defenses?  If you want to count
>> spam, it is mostly missing, at least as far as python-list is concerned.
>
> Its not a reward. Spammers are not like trolls, they don't hang around to
> see the result of their posts.

To me, the relevant difference is between posts related to python and 
those not.  It is usually clear which is which.

 > There no evidence at all that this Italian
> spammer is looking for replies or responses to his(?) posts. He apparently
> just fires them out.
>
> I think that it is relevant that comp.lang.python receives X spam messages
> from a certain person. It gives a picture of the health of the newsgroup:
> how much of it is spam? Hopefully only a small amount.

Python-list gets unrelated-to-python spam from lots of people.  They are 
not outliers (unlike jmf's now blocked trolls), but contaminents from a 
different universe.

I agree that the fraction of messages that are clearly spam has some 
interest in itself, and definitely should be as small as possible. But I 
contend that they should be excluded from a study of the universe of 
python-related messages.

My other point is that this small sliver that used to get passed through 
is extremely biased and statistically worthless as a study of 
python-list spamming.  If one wanted to study the rate and nature of 
contamination, or the effectiveness of filtering, one would need access 
to the raw stream of submissions.

-- 
Terry Jan Reedy



From alonso_oso at hotmail.com  Sat Dec 10 16:11:57 2016
From: alonso_oso at hotmail.com (=?iso-8859-1?Q?Alonso_L=F3pez?=)
Date: Sat, 10 Dec 2016 21:11:57 +0000
Subject: Can't find setuptools
Message-ID: <DB6P192MB0101C2E2E83D10598AFF83DFF3860@DB6P192MB0101.EURP192.PROD.OUTLOOK.COM>

Hi, I'm trying to install the package py-webrtcvad (github.com/wiseman/py-webrtcvad) and running into problems. Here's a summary of what I've tried so far:


?        Installed Python 3.5 and set up the Windows path environment so that it works from any directory.

?        Installed pip for Python.

?        Tried to install the package with python -m pip install webrtcvad, but it failed, returning the error "Unable to find vcvarsall.bat".

?        I found a blog that seems to deal with the problem vcvarsall.bat problem: blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/. Following the directions of that blog:

o   I installed Visual C++ Build Tools 2015. I tried running the program again straight away (without updating setuptools) and I received a lot of error messages which I didn't write down.

o   Following the directions in https://pypi.python.org/pypi/setuptools I removed the version of setuptools that came with my Python installation (v20), and installed the latest version (v30). This time the error message was "ImportError: No module named 'pip.utils.setuptools_build'".

o   Asked for assistance in the Python official chat. They made three suggestions:

?  Updating pip with python -m pip install --upgrade pip. Didn't work. Again, the error "ImportError: No module named 'pip.utils.setuptools_build'".

?  Reinstalling Visual C++ Build Tools 2015. No difference, same error again.

?  Reinstalling Python itself. The Python installer offers three choices: Repair, modify and uninstall.

?        Repair: Didn't work. Same error.

?        Modify: Doesn't look like it offers useful modifications for this.

?        Uninstall: Uninstalled and reinstalled. Still the same error.
I'm out of ideas. Can you help me?

Thank you,

Alonso L?pez



From vek.m1234 at gmail.com  Sun Dec 11 01:40:37 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sun, 11 Dec 2016 12:10:37 +0530
Subject: % string formatting - what special method is used for %d?
References: <o2g9f1$3oi$1@dont-email.me>
 <584bc233$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2isak$p09$1@dont-email.me>

Steve D'Aprano wrote:

> On Sat, 10 Dec 2016 06:06 pm, Veek M wrote:
> 
>> When we do:
>> 
>> print '%s %d' % ('hello', 10)
>> 
>> what special method is being invoked internally within the string-
>> format-specifier?
> 
> %d requires the argument to be an int, or able to be converted to int
> using the __int__ special method.
> 
> 
> py> class X(object):
> ...     def __int__(self):
> ...             return 42
> ...
> py> "%d" % X()
> '42'
> 
> 
> 
>> format() invokes format__
>> print invokes __str__
> 
> print actually invokes __str__ or __repr__, whichever is available.
> 
> 
> 
>> I'm basically trying to make sense of:
>> 
>> raise TypeError('urkle urkle %s' % list(dictionary))
>> <=> raise TypeError('urkle urkle %s' % [ key1, val1, key2, val2 ]
> 
> 
> The raise TypeError part of the code is irrelevant to your question.
> You should always simplify your code to only the part that is
> relevant.
> 
> raise TypeError(some_string)
> 
> behaves the same regardless of how some_string is made.
> 
> 
>> So, the % operator reads the format specifier and notices %s and
>> therefore calls __str__ in the list class to figure out how to
>> represent
>> [ key1, val1, key2, val2 ].
>> 
>> However what if I use %d? How do the other format specs work?
> 
> 
> The format specifiers are similar to these:
> 
> %s => str(obj), which ends up calling __str__ or __repr__
> 
> %r => repr(obj), which ends up calling __repr__ or __str__
> 
> %c => chr(obj), or obj must be a string of length 1
> 
> %d %i %u => int(obj), which ends up calling __int__
> 
> %x %X => int(obj), then convert to hexadecimal
> 
> %o => int(obj), then convert to octal
> 
> %e %E %f %g %G => float(obj), which ends up calling __float__
> 
> %% => a literal % sign
> 
> 
> 
> 
> 

Well take a look at this:
###########################################
#!/usr/bin/python

class Foo(int):
    def __init__(self, value):
        self.value = value
        
    def __str__(self):
        print '__str__'
        return str(self.value)
    
    def __int__(self):
        print '__int__'
        return self.value + 1
    

#'%s' % Foo(10) # %s is mapped to __str__
'%d' % Foo(20)
###########################################

here, '__str__' prints because when you do:
'%s' % x
the __str__ method is invoked. So internally %s invokes __str__ 
independent of print.

However the next line doesn't trigger any similar invocation with 
__int__ or__str__? (but int(Foo(10)) would invoked __int__)

Is there a way to trigger special methods using %d etc OR is this 
restricted to %s and why?










From bintacomputers at gmail.com  Sun Dec 11 02:16:53 2016
From: bintacomputers at gmail.com (Umar Yusuf)
Date: Sat, 10 Dec 2016 23:16:53 -0800 (PST)
Subject: Help in creating a dynamic/loop based on variables and CSV files
Message-ID: <bb8c8544-9ca4-4105-9b01-352988a64459@googlegroups.com>

Hi all,
I need your help with any of these questions?

1- http://stackoverflow.com/questions/41083699/python-create-dynamic-loop-based-on-variables-and-csv

2- http://stackoverflow.com/questions/41081800/python-pandas-how-to-use-dataframe-cell-to-search-another-dataframe-column-and

Thanks in advance for your time.


From ian.g.kelly at gmail.com  Sun Dec 11 02:21:18 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sun, 11 Dec 2016 00:21:18 -0700
Subject: % string formatting - what special method is used for %d?
In-Reply-To: <o2isak$p09$1@dont-email.me>
References: <o2g9f1$3oi$1@dont-email.me>
 <584bc233$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o2isak$p09$1@dont-email.me>
Message-ID: <CALwzid=5hbK3yFdVJ6G-5zoip363wn0T4naFNZHbgmGyxyxc+w@mail.gmail.com>

On Sat, Dec 10, 2016 at 11:40 PM, Veek M <vek.m1234 at gmail.com> wrote:
> Well take a look at this:
> ###########################################
> #!/usr/bin/python
>
> class Foo(int):
>     def __init__(self, value):
>         self.value = value
>
>     def __str__(self):
>         print '__str__'
>         return str(self.value)
>
>     def __int__(self):
>         print '__int__'
>         return self.value + 1
>
>
> #'%s' % Foo(10) # %s is mapped to __str__
> '%d' % Foo(20)
> ###########################################
>
> here, '__str__' prints because when you do:
> '%s' % x
> the __str__ method is invoked. So internally %s invokes __str__
> independent of print.
>
> However the next line doesn't trigger any similar invocation with
> __int__ or__str__? (but int(Foo(10)) would invoked __int__)

This is probably because Foo inherits from int. Foo(20) is already an
int so there is no conversion to be done; Python simply uses the int
value and ignores the __int__ method in this case.

> Is there a way to trigger special methods using %d etc OR is this
> restricted to %s and why?

For an object that is already an int, probably not.

However you may want to revisit your decision to make Foo inherit from
int and question whether that is really sensible if you're also
wanting to override the __int__ method. What does that mean if
something is an int but also provides a method to convert to int? It's
a contradiction.


From vek.m1234 at gmail.com  Sun Dec 11 03:52:39 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sun, 11 Dec 2016 14:22:39 +0530
Subject: % string formatting - what special method is used for %d?
References: <o2g9f1$3oi$1@dont-email.me>
 <584bc233$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o2isak$p09$1@dont-email.me>
 <CALwzid=5hbK3yFdVJ6G-5zoip363wn0T4naFNZHbgmGyxyxc+w@mail.gmail.com>
 <mailman.5.1481440927.2322.python-list@python.org>
Message-ID: <o2j425$c0l$1@dont-email.me>

Ian Kelly wrote:

> On Sat, Dec 10, 2016 at 11:40 PM, Veek M <vek.m1234 at gmail.com> wrote:
>> Well take a look at this:
>> ###########################################
>> #!/usr/bin/python
>>
>> class Foo(int):
>>     def __init__(self, value):
>>         self.value = value
>>
>>     def __str__(self):
>>         print '__str__'
>>         return str(self.value)
>>
>>     def __int__(self):
>>         print '__int__'
>>         return self.value + 1
>>
>>
>> #'%s' % Foo(10) # %s is mapped to __str__
>> '%d' % Foo(20)
>> ###########################################
>>
>> here, '__str__' prints because when you do:
>> '%s' % x
>> the __str__ method is invoked. So internally %s invokes __str__
>> independent of print.
>>
>> However the next line doesn't trigger any similar invocation with
>> __int__ or__str__? (but int(Foo(10)) would invoked __int__)
> 
> This is probably because Foo inherits from int. Foo(20) is already an
> int so there is no conversion to be done; Python simply uses the int
> value and ignores the __int__ method in this case.
> 
>> Is there a way to trigger special methods using %d etc OR is this
>> restricted to %s and why?
> 
> For an object that is already an int, probably not.
> 
> However you may want to revisit your decision to make Foo inherit from
> int and question whether that is really sensible if you're also
> wanting to override the __int__ method. What does that mean if
> something is an int but also provides a method to convert to int? It's
> a contradiction.

Ah! thanks guys, now it works great. (I still need to ponder all this 
anyhow but yay!)


From __peter__ at web.de  Sun Dec 11 05:33:03 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 11 Dec 2016 11:33:03 +0100
Subject: Help in creating a dynamic/loop based on variables and CSV files
References: <bb8c8544-9ca4-4105-9b01-352988a64459@googlegroups.com>
Message-ID: <o2ja0v$j80$1@blaine.gmane.org>

Umar Yusuf wrote:

> Hi all,
> I need your help with any of these questions?
> 
> 1-
> http://stackoverflow.com/questions/41083699/python-create-dynamic-loop-based-on-variables-and-csv

You should really make a serious attempt to explain the problem in plain 
english. Throwing a piece of code at other people is usually not enough to 
get a meaningful answer.

That said, my crystal ball tells me that you want to remove all rows of a 
csv before the one containing a keyword in the first column. You can do it 
for one keyword/csv-infile/csv-outfile triple,

> with open("keywords.txt", "rb") as keywords, open('folding umbrella-sort-
highest.csv', 'rb') as g, open('lego minecraft-sort-highest.csv', 'rb') as 
f, open('filename1.csv', 'wb') as myfile1, open('filename2.csv', 'wb') as 
myfile2:
> 
> # Step1: Read contents of keywords.tex in variables
>     ky = list(keywords.readlines())
>     ky1, ky2 = ky[0], ky[1]
> 
> # Step2: Read and process folding umbrella-sort-highest.csv
>     reader = csv.reader(g)
>     umbrella_list = list(reader)
>     list1 = filter(lambda e: e[0] in ky1, umbrella_list)
> 
>     list2 = list(chain(*list1))
>     # or better: (available since Python 2.6)
>     # print list(chain.from_iterable(list1))
> 
>     ind_prt1 = umbrella_list.index(list2) +1 
>     mylist1 = umbrella_list[:ind_prt1]
> 
>     for r in mylist1:
>         wr = csv.writer(myfile1, quoting=csv.QUOTE_ALL)
>         wr.writerow(r)
> 

and by employing copy-and-paste coding you have managed to do it for two 
keywords. Now you want to generalize it for an arbitrary number of triples.

The usual approach is to put the code that works for one instance into a 
function and call that for every set of arguments

def process_one(keyword, infile, outfile):
    # your code

for keyword, infile, outfile in triples:
    process_one(keyword, infile, outfile)

Where can you get those triples? Rather than putting just the keywords into 
a text file i would use a csv with three columns:

first_keyword,folding umbrella-sort-highest.csv,filename1.csv
second_keyword,lego minecraft-sort-highest.csv,filename2.csv
...

Then the calling code becomes

with open("keywords.csv", "rb") as f:
    for keyword, infile, outfile in csv.reader(f):
        process_one(keyword, infile, outfile)

Now to the contents of process_one(). Your code is very complicated. If it 
does what I think it can be rewritten as

def process_one(keyword, infile, outfile):
    with open(outfile, "wb") as outstream:
        writer = csv.writer(outstream, csv.QUOTE_ALL)
        with open(infile, "rb") as instream:
            for row in csv.reader(instream):
                writer.writerow(row)
                if row[0] in keyword:
                    break

By the way, the

row[0] in keyword

test looks odd. Should that be

keyword in row[0]

or

keyword == row[0]

? Without a proper description of the problem that led to your code I have 
no way to tell.

> 2-
> http://stackoverflow.com/questions/41081800/python-pandas-how-to-use-dataframe-cell-to-search-another-dataframe-column-and
> 
> Thanks in advance for your time.




From darcy at PyGreSQL.org  Sun Dec 11 06:31:04 2016
From: darcy at PyGreSQL.org (D'Arcy Cain)
Date: Sun, 11 Dec 2016 06:31:04 -0500
Subject: PyGreSQL 5.0.3 released
Message-ID: <ef54ed94-44e0-2a43-d460-cc0755cd2a10@PyGreSQL.org>

Release 5.0.3 of PyGreSQL.

It is available at: http://pygresql.org/files/PyGreSQL-5.0.3.tar.gz.

If you are running NetBSD, look in the packages directory under databases.
There is also a package in the FreeBSD ports collection.

Please refer to the changelog.txt file for things that have changed in 
this version.

Please refer to `readme.txt for general information.

This version has been built and unit tested on:
  - NetBSD
  - FreeBSD
  - openSUSE
  - Ubuntu
  - Windows 7 with both MinGW and Visual Studio
  - PostgreSQL 9.0 to 9.5 32 and 64bit
  - Python 2.6, 2.7, 3.3, 3.4 and 3.5 32 and 64bit

-- 
D'Arcy J.M. Cain
PyGreSQL Development Group
http://www.PyGreSQL.org IM:darcy at Vex.Net


From best_lay at yahoo.com  Sun Dec 11 11:02:56 2016
From: best_lay at yahoo.com (Wildman)
Date: Sun, 11 Dec 2016 10:02:56 -0600
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>
 <o2he34$kbt$1@dont-email.me>
Message-ID: <c6KdnU5S5oEt5dDFnZ2dnUU7-fudnZ2d@giganews.com>

On Sat, 10 Dec 2016 12:31:33 -0500, DFS wrote:

> 

After correcting my stupid oversights, the code runs fine
up to the point where the user agents are printed.  I get
an error saying that 'User-Agent' is an unsupported header
field.  It must have something to do with giganews.  If I
use aioe.org I don't get the error and the user agents are
printed.

I don't think it is a problem with the code but any thoughts
why giganews is not playing nice?  And it is not related to
the python group.  I have tried on other groups and i get
the same error.  Here is the complete error message.

Traceback (most recent call last):
  File "./nntp.py", line 27, in <module>
    printStat("User-Agent","User-Agents",m)
  File "./nntp.py", line 12, in printStat
    r,d=n.xhdr(st,'%s-%s'%rg)
  File "/usr/lib/python2.7/nntplib.py", line 470, in xhdr
    resp, lines = self.longcmd('XHDR ' + hdr + ' ' + str, file)
  File "/usr/lib/python2.7/nntplib.py", line 273, in longcmd
    return self.getlongresp(file)
  File "/usr/lib/python2.7/nntplib.py", line 244, in getlongresp
    resp = self.getresp()
  File "/usr/lib/python2.7/nntplib.py", line 229, in getresp
    raise NNTPPermanentError(resp)
nntplib.NNTPPermanentError: 501 unsupported header field

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From wanderer at dialup4less.com  Sun Dec 11 11:28:30 2016
From: wanderer at dialup4less.com (Wanderer)
Date: Sun, 11 Dec 2016 08:28:30 -0800 (PST)
Subject: pySerial raw data
Message-ID: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>

I have an outdoor thermometer that transmits to an indoor receiver at 433Mhz. I also have a 433Mhz USB serial port jig from a TI development tool. I would like to use the TI USB serial port to capture the temperature information. The TI USB port registers as a COM port that I can access with pySerial. Now the datasheet from the temperature probe only says that the RF frequency is 433MHz and that it transmits every 39 seconds. Since I don't know what protocol the thermometer uses or baud rate, I want to look at the rawest level of data collected with the USB com port and see if I can make anything out of the gobbledy gook coming in. Is there a way to get this kind of data from pySerial? I've tried scanning at different baud rates but so far I haven't captured anything.

Also in the advanced settings in windows device manager, there are some settings for Fifo buffers, and receive and transmit buffers. Can these be accessed in pySerial? Does pySerial override the settings for baud rate, etc in windows device manager or do I need to set those to match what I'm using in pySerial?

Thanks


From python at mrabarnett.plus.com  Sun Dec 11 12:51:43 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 11 Dec 2016 17:51:43 +0000
Subject: pySerial raw data
In-Reply-To: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>
References: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>
Message-ID: <6f1b785d-1178-95b5-f316-30d51caf337e@mrabarnett.plus.com>

On 2016-12-11 16:28, Wanderer wrote:
> I have an outdoor thermometer that transmits to an indoor receiver at 433Mhz. I also have a 433Mhz USB serial port jig from a TI development tool. I would like to use the TI USB serial port to capture the temperature information. The TI USB port registers as a COM port that I can access with pySerial. Now the datasheet from the temperature probe only says that the RF frequency is 433MHz and that it transmits every 39 seconds. Since I don't know what protocol the thermometer uses or baud rate, I want to look at the rawest level of data collected with the USB com port and see if I can make anything out of the gobbledy gook coming in. Is there a way to get this kind of data from pySerial? I've tried scanning at different baud rates but so far I haven't captured anything.
>
> Also in the advanced settings in windows device manager, there are some settings for Fifo buffers, and receive and transmit buffers. Can these be accessed in pySerial? Does pySerial override the settings for baud rate, etc in windows device manager or do I need to set those to match what I'm using in pySerial?
>
What is the make and model of the thermometer? Is the datasheet online 
somewhere?



From juan0christian at gmail.com  Sun Dec 11 15:10:56 2016
From: juan0christian at gmail.com (Juan C.)
Date: Sun, 11 Dec 2016 18:10:56 -0200
Subject: The right way to 'call' a class attribute inside the same class
Message-ID: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>

I'm watching a Python course and was presented a topic regarding classes.
One of the examples were:

box.py

class Box:
    serial = 100

    def __init__(self, from_addr, to_addr):
        self.from_addr = from_addr
        self.to_addr = to_addr
        self.serial = Box.serial
        Box.serial += 1


from box import *

a = Box('19 Beech Ave. Seattle, WA 98144', '49 Carpenter Street North
Brunswick, NJ 08902')
b = Box('68 N. Church Dr. Vicksburg, MS 39180', '8 Lake Forest Road
Princeton, NJ 08540')

print(a.serial)  # print: 100
print(b.serial)  # print: 101
print(Box.serial)  # print: 102


The instructor said that the right way to call a class attribute is to use
'Class.class_attr' notation, but on the web I found examples where people
used 'self.class_attr' to call class attributes. I believe that using the
first notation is better ('Class.class_attr'), this way the code is more
explicit, but is there any rules regarding it?


From jon+usenet at unequivocal.eu  Sun Dec 11 15:26:47 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sun, 11 Dec 2016 20:26:47 -0000 (UTC)
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>
 <o2he34$kbt$1@dont-email.me>
 <c6KdnU5S5oEt5dDFnZ2dnUU7-fudnZ2d@giganews.com>
Message-ID: <slrno4rdmp.298.jon+usenet@sable.unequivocal.eu>

On 2016-12-11, Wildman <best_lay at yahoo.com> wrote:
> I don't think it is a problem with the code but any thoughts
> why giganews is not playing nice?

Most likely because you're calling XHDR on a header which is not in
the server's overview file.


From wanderer at dialup4less.com  Sun Dec 11 16:29:38 2016
From: wanderer at dialup4less.com (Wanderer)
Date: Sun, 11 Dec 2016 13:29:38 -0800 (PST)
Subject: pySerial raw data
In-Reply-To: <mailman.7.1481478713.2322.python-list@python.org>
References: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>
 <6f1b785d-1178-95b5-f316-30d51caf337e@mrabarnett.plus.com>
 <mailman.7.1481478713.2322.python-list@python.org>
Message-ID: <d16bef77-9cc6-4a94-87b1-fa3c27260ebe@googlegroups.com>

On Sunday, December 11, 2016 at 12:52:04 PM UTC-5, MRAB wrote:
> On 2016-12-11 16:28, Wanderer wrote:
> > I have an outdoor thermometer that transmits to an indoor receiver at 433Mhz. I also have a 433Mhz USB serial port jig from a TI development tool. I would like to use the TI USB serial port to capture the temperature information. The TI USB port registers as a COM port that I can access with pySerial. Now the datasheet from the temperature probe only says that the RF frequency is 433MHz and that it transmits every 39 seconds. Since I don't know what protocol the thermometer uses or baud rate, I want to look at the rawest level of data collected with the USB com port and see if I can make anything out of the gobbledy gook coming in. Is there a way to get this kind of data from pySerial? I've tried scanning at different baud rates but so far I haven't captured anything.
> >
> > Also in the advanced settings in windows device manager, there are some settings for Fifo buffers, and receive and transmit buffers. Can these be accessed in pySerial? Does pySerial override the settings for baud rate, etc in windows device manager or do I need to set those to match what I'm using in pySerial?
> >
> What is the make and model of the thermometer? Is the datasheet online 
> somewhere?

http://global.oregonscientific.com/manual/THN132N.pdf


From no.email at nospam.invalid  Sun Dec 11 16:36:58 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Sun, 11 Dec 2016 13:36:58 -0800
Subject: pySerial raw data
References: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>
Message-ID: <87inqq2lmt.fsf@nightsong.com>

Wanderer <wanderer at dialup4less.com> writes:
> I also have a 433Mhz USB serial port jig from a TI development
> tool.... The TI USB port registers as a COM port that I can access
> with pySerial.

If the TI jig has 433 mhz (LORA?) at one end and serial at the other,
you have to find the port parameters in the docs for the TI jig, not the
thermometer.

If you don't have docs you can often figure out the right settings by
trial and error.  If not, the direct approach is to use an oscilloscope.


From python at mrabarnett.plus.com  Sun Dec 11 16:52:38 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 11 Dec 2016 21:52:38 +0000
Subject: pySerial raw data
In-Reply-To: <d16bef77-9cc6-4a94-87b1-fa3c27260ebe@googlegroups.com>
References: <2e42a1b4-513a-43f3-bb0d-3596822d572d@googlegroups.com>
 <6f1b785d-1178-95b5-f316-30d51caf337e@mrabarnett.plus.com>
 <mailman.7.1481478713.2322.python-list@python.org>
 <d16bef77-9cc6-4a94-87b1-fa3c27260ebe@googlegroups.com>
Message-ID: <6dc93d02-e912-824e-6b4b-8a90e6888872@mrabarnett.plus.com>

On 2016-12-11 21:29, Wanderer wrote:
> On Sunday, December 11, 2016 at 12:52:04 PM UTC-5, MRAB wrote:
>> On 2016-12-11 16:28, Wanderer wrote:
>> > I have an outdoor thermometer that transmits to an indoor receiver at 433Mhz. I also have a 433Mhz USB serial port jig from a TI development tool. I would like to use the TI USB serial port to capture the temperature information. The TI USB port registers as a COM port that I can access with pySerial. Now the datasheet from the temperature probe only says that the RF frequency is 433MHz and that it transmits every 39 seconds. Since I don't know what protocol the thermometer uses or baud rate, I want to look at the rawest level of data collected with the USB com port and see if I can make anything out of the gobbledy gook coming in. Is there a way to get this kind of data from pySerial? I've tried scanning at different baud rates but so far I haven't captured anything.
>> >
>> > Also in the advanced settings in windows device manager, there are some settings for Fifo buffers, and receive and transmit buffers. Can these be accessed in pySerial? Does pySerial override the settings for baud rate, etc in windows device manager or do I need to set those to match what I'm using in pySerial?
>> >
>> What is the make and model of the thermometer? Is the datasheet online
>> somewhere?
>
> http://global.oregonscientific.com/manual/THN132N.pdf
>
That datasheet says """This product is compatible with various wireless 
weather station products."""

OK, so that suggests that there's a standard of some kind somewhere.

Googling for """wireless weather station protocol""" gives:

Reverse engineering wireless weather stations
hackaday.com/2011/06/13/reverse-engineering-wireless-weather-stations/

and that page leads to:

TX29 Protocol
http://fredboboss.free.fr/articles/tx29.php

Good luck!



From rosuav at gmail.com  Sun Dec 11 17:26:41 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 12 Dec 2016 09:26:41 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
Message-ID: <CAPTjJmqfsH22DQ4+BT1Kc50CK-yyrN60xE6MwJXLzf0FbfV--Q@mail.gmail.com>

On Mon, Dec 12, 2016 at 7:10 AM, Juan C. <juan0christian at gmail.com> wrote:
> class Box:
>     serial = 100
>
>     def __init__(self, from_addr, to_addr):
>         self.from_addr = from_addr
>         self.to_addr = to_addr
>         self.serial = Box.serial
>         Box.serial += 1
>

I would say that this is awkward usage; the class attribute isn't
being used as a default for the instance, it's being used as "the next
one". I would rename the class attribute to "next_serial". That would
give you the freedom to use whichever notation you like, as there
won't be a conflict.

ChrisA


From greg.ewing at canterbury.ac.nz  Sun Dec 11 17:29:19 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 12 Dec 2016 11:29:19 +1300
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <mailman.16.1481487105.2322.python-list@python.org>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
Message-ID: <eb62a1FoldnU1@mid.individual.net>

Juan C. wrote:
> The instructor said that the right way to call a class attribute is to use
> 'Class.class_attr' notation, but on the web I found examples where people
> used 'self.class_attr' to call class attributes. I believe that using the
> first notation is better ('Class.class_attr'), this way the code is more
> explicit, but is there any rules regarding it?

It depends on how the class attribute is being used.

If you're only reading the attribute, either way will work.
Which one is more appropriate depends on what the attribute
is used for. Often a class attribute is used as a default
value for an instance attribute, in which case accessing it
via the instance is entirely appropriate. On the other
hand, if it's truly mean to be an attribute of the class
itself, accessing it via the class is probably clearer.

If the attribute is being written, you don't have any
choice. If you want to rebind the attribute in the class,
you have to access it via the class. This is the case
for this line in your example:

         Box.serial += 1

If instead you did 'self.serial += 1' it would create
a new instance attribute shadowing the class attribute,
and the class attribute would remain bound to its
previous value.

-- 
Greg


From tjreedy at udel.edu  Sun Dec 11 19:51:05 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 11 Dec 2016 19:51:05 -0500
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <eb62a1FoldnU1@mid.individual.net>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <eb62a1FoldnU1@mid.individual.net>
Message-ID: <o2ks9l$ubs$1@blaine.gmane.org>

On 12/11/2016 5:29 PM, Gregory Ewing wrote:
> Juan C. wrote:
>> The instructor said that the right way to call a class attribute is to
>> use
>> 'Class.class_attr' notation, but on the web I found examples where people
>> used 'self.class_attr' to call class attributes. I believe that using the
>> first notation is better ('Class.class_attr'), this way the code is more
>> explicit, but is there any rules regarding it?

Yes. Use the form appropriate to the situation.  In other words, use 
open-eyed rule, not a closed-eye rule.  This applies to much of Python 
programming and programming is general.  Greg nicely explains the 
application of this rule.

> It depends on how the class attribute is being used.
>
> If you're only reading the attribute, either way will work.
> Which one is more appropriate depends on what the attribute
> is used for. Often a class attribute is used as a default
> value for an instance attribute, in which case accessing it
> via the instance is entirely appropriate.

The use of a (constant) class attribute as default instance attribute 
might be an optimization added after the first version of the class, or 
one that could disappear in the future.

 > On the other
> hand, if it's truly mean to be an attribute of the class
> itself, accessing it via the class is probably clearer.
>
> If the attribute is being written, you don't have any
> choice. If you want to rebind the attribute in the class,
> you have to access it via the class. This is the case
> for this line in your example:
>
>         Box.serial += 1
>
> If instead you did 'self.serial += 1' it would create
> a new instance attribute shadowing the class attribute,
> and the class attribute would remain bound to its
> previous value.

I agree with the other post suggesting using 'next_serial' as the class 
attribute, as that is what the class attribute is.  I would access it as 
Box.serial.

Instance methods should normal be accessed through an instance, though 
there are exceptions.

-- 
Terry Jan Reedy



From steve+python at pearwood.info  Sun Dec 11 20:34:04 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 12 Dec 2016 12:34:04 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
Message-ID: <584dfe8e$0$1602$c3e8da3$5496439d@news.astraweb.com>

On Mon, 12 Dec 2016 07:10 am, Juan C. wrote:

> I'm watching a Python course and was presented a topic regarding classes.
> One of the examples were:
> 
> box.py
> 
> class Box:
>     serial = 100
> 
>     def __init__(self, from_addr, to_addr):
>         self.from_addr = from_addr
>         self.to_addr = to_addr
>         self.serial = Box.serial
>         Box.serial += 1


If you want to distinguish between an instance attribute and a class
attribute, you must specify the instance or the class:

    self.serial # may be the instance attribute, or the class attribute
    Box.serial  # always the class attribute


But what happens inside a subclass?

class BoxWithLid(Box):
    pass

Generally we expect methods called from the subclass BoxWithLid to refer to
the BoxWithLid attribute, not the Box attribute. So Box.serial will be
wrong when the method is called from a subclass.

    type(self).serial  # still correct when called from a subclass

> The instructor said that the right way to call a class attribute is to use
> 'Class.class_attr' notation, 

That is nearly always wrong, since it will break when you subclass. If you
do that, you should document that the class is not expected to be
subclasses, and may not work correctly if you do.


> but on the web I found examples where people 
> used 'self.class_attr' to call class attributes. I believe that using the
> first notation is better ('Class.class_attr'), this way the code is more
> explicit, but is there any rules regarding it?

Assignment to self.serial will always create or affect an instance
attribute. But merely retrieving self.serial will use inheritance to try
returning the instance attribute *if it exists*, and if not, fall back on
the class attribute (or a superclass).

This is especially useful for read-only defaults, constants or configuration
settings.


class Document:
    page_size = A4

    def page_area(self):
        dimensions = list(self.page_size)
        dimensions[0] -= self.left_margin
        dimensions[1] -= self.right_margin
        dimensions[2] -= self.top_margin
        dimensions[3] -= self.bottom_margin
        return dimensions


doc = Document()  # uses the default page size of A4

doc.page_size = Foolscape  # override the default


It is a matter of taste and context whether you do this, or the more
conventional way:

class Document:
    def __init__(self):
        self.page_size = A4


Use whichever is better for your specific class.


So... in summary:


When *assigning* to an attribute:

- use `self.attribute = ...` when you want an instance attribute;

- use `Class.attribute = ...` when you want a class attribute in 
  the same class regardless of which subclass is being used;

- use `type(self).attribute = ...` when you want a class attribute
  in a subclass-friendly way.


When *retrieving* an attribute:

- use `self.attribute` when you want to use the normal inheritance
  rules are get the instance attribute if it exists, otherwise a
  class or superclass attribute;

- use `type(self).attribute` when you want to skip the instance
  and always return the class or superclass attribute.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From best_lay at yahoo.com  Sun Dec 11 23:57:33 2016
From: best_lay at yahoo.com (Wildman)
Date: Sun, 11 Dec 2016 22:57:33 -0600
Subject: CLP stats: last 500 posts
References: <o2f6bn$70u$2@dont-email.me>
 <StGdnbCBvde3q9HFnZ2dnUU7-cGdnZ2d@giganews.com>
 <o2he34$kbt$1@dont-email.me>
 <c6KdnU5S5oEt5dDFnZ2dnUU7-fudnZ2d@giganews.com>
 <o2k0ps$75d$2@dont-email.me>
Message-ID: <M4udnWsptOqgs9PFnZ2dnUU7-fWdnZ2d@giganews.com>

On Sun, 11 Dec 2016 12:03:07 -0500, DFS wrote:

> For this short stat version I only used the 'User-Agent' header.  I have 
> a longer version that uses both 'User-Agent' and 'X-Newsreader'
> 
> 
> You can put a conditional in place for now:
> 
> if s='giganews':
> 	printStat("X-Newsreader","News Readers",m)
> else:
> 	printStat("User-Agent","User-Agents",m)

Thanks but I had already tried X-Newsreader and I got the
same result.  It is odd because if you look at my headers
there is an entry for User-Agent....

User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
 git://git.gnome.org/pan2; x86_64-pc-linux-gnu)

<scratching head>

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From PointedEars at web.de  Mon Dec 12 09:34:27 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Mon, 12 Dec 2016 15:34:27 +0100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
Message-ID: <3924532.8F6SAcFxjW@PointedEars.de>

Juan C. wrote:

> I'm watching a Python course and was presented a topic regarding classes.
> One of the examples were:
> 
> box.py
> 
> class Box:
>     serial = 100
> 
>     def __init__(self, from_addr, to_addr):
>         self.from_addr = from_addr
>         self.to_addr = to_addr
>         self.serial = Box.serial
>         Box.serial += 1
> 
> 
> from box import *
> 
> a = Box('19 Beech Ave. Seattle, WA 98144', '49 Carpenter Street North
> Brunswick, NJ 08902')
> b = Box('68 N. Church Dr. Vicksburg, MS 39180', '8 Lake Forest Road
> Princeton, NJ 08540')
> 
> print(a.serial)  # print: 100
> print(b.serial)  # print: 101
> print(Box.serial)  # print: 102
> 
> 
> The instructor said that the right way to call a class attribute is to use
> 'Class.class_attr' notation, but on the web I found examples where people
> used 'self.class_attr' to call class attributes. I believe that using the
> first notation is better ('Class.class_attr'), this way the code is more
> explicit, but is there any rules regarding it?

First of all, the proper term for what you are doing there is _not_ ?call?;  
you are _accessing_ an attribute instead.

To call something means generally in programming, and in Python, to execute 
it as a function instead: In the code above, the class object referred to by 
?Box? is called twice in order to instantiate twice (calling a class object 
in Python means to instantiate it, implicitly calling its constructor 
method, ?__init__?, if any), and the global ?print? function is called three 
times.

Second, you do not appear to be aware that the notations

  C.foo

and

  self.foo

within a method of the class object referred to by ?C? are _not_ equivalent:

?C? is _not_ a "more explicit" way to access the class than ?self?; it is 
referring to a *different* (type of) *object* instead, a *class* object.  
(This is different in Python than in, for example, PHP.)

Assignment to ?C.foo? modifies an attribute named ?foo? of the class object 
referred to ?C?, i.e. it modifies the value of that attribute *for all 
instances of that class* (but see below).

Assignment to ?self.foo? modifies only the ?foo? attribute *of a particular 
instance* of the class, or adds an attribute of that name to it; namely, the 
instance on which the method was called that modifies/adds the ?foo? 
attribute (referred to by the formal parameter ?self? of the method?).

$ python3 -c '
class C:
    foo = 23

    def bar (self):
        self.foo = 42

    def baz (self):
        C.foo = 42

print(C.foo)
o = C()
o2 = C()           
print(C.foo, o.foo, o2.foo)
o.bar()
print(C.foo, o.foo, o2.foo)
o.baz()
print(C.foo, o.foo, o2.foo)
'
23
23 23 23
23 42 23
42 42 42


To illustrate, the (simplified) UML diagrams? for the state of the program 
after each relevant executed chunk of code (as always recommended, use a 
fixed-width font for display):

#--------------------
class C:
    foo = 23

    def bar (self):
        self.foo = 42

    def baz (self):
        C.foo = 42

#--------------------

              ,-----------------.
    C ------> : :type           :
              :=================:
              : +foo : int = 23 :
              :-----------------:
              : +bar()          :
              : +baz()          :
              `-----------------'

#--------------------
o = C()
#--------------------

              ,-----------------.
    C ------> : :type           :
              :=================:
              : +foo : int = 23 :
              :-----------------:
              : +bar()          :
              : +baz()          :
              `-----------------'
                       ^
                       :
                ,-------------.
    o --------> : :__main__.C :
                :-------------:
                `-------------'

#--------------------
o2 = C()           
#--------------------

                ,-------------------.
    C --------> : :type             :
                :===================:
                : +foo : int = 23   :
                :-------------------:
                : +bar()            :
                : +baz()            :
                `-------------------'
                  ^               ^                   
                  :               :
           ,-------------. ,-------------.
    o ---> : :__main__.C : : :__main__.C : <--- o2
           :-------------: :-------------:
           `-------------' `-------------'

#--------------------
o.bar()
#--------------------

                ,-------------------------.
    C --------> : :type                   :
                :=========================:
                : +foo : int = 23         :
                :-------------------------:
                : +bar()                  :
                : +baz()                  :
                `-------------------------'
                    ^                 ^
                    :                 :
           ,-----------------. ,-------------.
    o ---> : :__main__.C     : : :__main__.C : <--- o2
           :-----------------: :-------------:
           : +foo : int = 42 : `-------------'
           `-----------------' 

#--------------------
o.baz()
#--------------------

                ,-------------------------.
    C --------> : :type                   :
                :=========================:
                : +foo : int = 42         :
                : +bar()                  :
                : +baz()                  :
                `-------------------------'
                    ^                 ^
                    :                 :
           ,-----------------. ,-------------.
    o ---> : :__main__.C     : : :__main__.C : <--- o2
           :-----------------: :-------------:
           : +foo : int = 42 : `-------------'
           `-----------------' 

Note that (AIUI) in this example the instances of the class referred by ?C? 
do not have an *own* ?foo? property in the beginning, so until bar() is 
called on them, they inherit that property (and its value) from that class.?

IOW, only as long as the instance on which a method is called does not have 
an *own* attribute of name ?foo? are the notations ?C.foo? and ?self.foo?, 
where C is the class of that instance, equivalent in that method.

Since you cannot be certain without testing the instance and the class (and 
that would be less efficient), it is prudent to refer to class attributes 
through the class name and to attributes of instances through ?self?.

________
?  It should be noted that (different to, e.g., PHP), the ?self? in Python 
   is not a predefined name, but just a code convention.  In fact, if you
   call a method of a class on one of its instances, the method is
   implicitly passed a reference to the instance as the first argument. 
   Therefore,

     class C:
         foo = 23

         def bar (x)
             x.foo = 42

     o = C()
     o.bar()

   is syntactically valid and functionally equivalent to

     class C:
         foo = 23

         def bar (self)
             self.foo = 42

     o = C()
     o.bar()

?  Legend:  X ---> Y   The value of X is a reference to Y

               Y
               ^
               :       X inherits from Y
               X

               +X      X is public
                X()    X is a method
               :X      The (return) type (of the preceding) is X
              X = Y    The current value of X is Y

           ,--------.
           : X      :  X is a class
           :========:
           `--------'

           ,--------.
           : X      :  X is an instance of a class
           :--------:
           `--------'

?  How can one tell the difference in Python between a pre-initialized, 
   inherited attribute value and one own that is just equal to the inherited
   one?  In ECMAScript, this.hasOwnProperty("foo") would return ?false? if
   the property were inherited, ?true? otherwise.  But in Python,
   hasattr(self, "foo") returns ?True? regardless whether the ?foo? 
   attribute of the calling instance has been assigned a value explicitly.
   What is the Python equivalent of ECMAScript?s
   Object.prototype.hasOwnProperty() method?
-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From PointedEars at web.de  Mon Dec 12 09:44:25 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Mon, 12 Dec 2016 15:44:25 +0100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
Message-ID: <1931692.72vocr9iq0@PointedEars.de>

Thomas 'PointedEars' Lahn wrote:

> Note that (AIUI) in this example the instances of the class referred by
> ?C? do not have an *own* ?foo? property in the beginning, so until bar()
> is called on them, they inherit that property (and its value) from that
> class.?

For proper *Python* terminology, s/property/attribute/g here (see below).

> [?]
> ________
> [?]
> ?  How can one tell the difference in Python between a pre-initialized,
>    inherited attribute value and one own that is just equal to the
>    inherited one?  In ECMAScript, this.hasOwnProperty("foo") would return
>    ?false? if the property were inherited, ?true? otherwise.  But in
>    Python, hasattr(self, "foo") returns ?True? regardless whether the
>    ?foo? attribute of the calling instance has been assigned a value
>    explicitly.  What is the Python equivalent of ECMAScript?s
>    Object.prototype.hasOwnProperty() method?

-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From dr.roman.graf at gmail.com  Mon Dec 12 10:27:20 2016
From: dr.roman.graf at gmail.com (roma)
Date: Mon, 12 Dec 2016 07:27:20 -0800 (PST)
Subject: Django broken pipe error
In-Reply-To: <mailman.132.1481130930.2253.python-list@python.org>
References: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
 <843bd5a8-f127-4fdc-8817-0b1645c1872d@googlegroups.com>
 <CAO1D73H21KK14yheCLdH+LjXH_YWMaOoDM5Q7qDAm8kq1cS9QA@mail.gmail.com>
 <mailman.132.1481130930.2253.python-list@python.org>
Message-ID: <e246f8df-bead-4ae4-b41e-f18f2c1ba2bb@googlegroups.com>

On Wednesday, December 7, 2016 at 6:15:41 PM UTC+1, justin walters wrote:
> On Wed, Dec 7, 2016 at 1:08 AM, <dr.roman.graf at gmail.com> wrote:
> 
> > Thank you Justin,
> >
> > I'm on the dev server and should present results in this way.
> >
> > Yes, I use manage.py runserver --insecure to start the server (from
> > PyCharm).
> >
> > My views.py call:
> >
> > @detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
> > def export_report(request):
> >     body_unicode = request.body.decode('utf-8')
> >     body_str = body_unicode.encode('ascii','ignore')
> >     attr_list = body_str.split('&')
> >     attr_dict = {}
> >     if (len(attr_list) > 0):
> >         for attr in attr_list:
> >            ...
> >            key = key_value_pair[0]
> >            attr_dict[key] = key_value_pair[1]
> >     trend = trends.calculate_trend(
> >         attr_dict['search_phrase']
> >         , attr_dict['time_from']
> >         , attr_dict['time_to']
> >         , attr_dict['time_scale']
> >     )
> >     attr_dict['trend'] = trend
> >     res = str(json.dumps(attr_dict))
> >     return HttpResponse(res, content_type="text/plain; charset=utf-8")
> >
> > and trend calculation in trends.py with database calls:
> >
> > def calculate_trend(query_phrase, time_from, time_to, time_scale):
> >     # check in database if trend already exists
> >     try:
> >         db_trend = Trend.objects.get(pk=query_phrase)
> >         if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
> >                 and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
> >                 and db_trend.granularity == time_scale:
> >             logger.info("trend already exists.")
> >             existing_trend_dict = ast.literal_eval(db_trend.content)
> >             return existing_trend_dict
> >     except Trend.DoesNotExist:
> >         logger.info("It is a new trend search.")
> >     trend_dict = {}
> >     start_time = pd.Timestamp(value[0])
> >     end_time = pd.Timestamp(value[-1])
> >     freq = ... get frequency using pandas lib
> >     trend_dict[key] = freq
> >     json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
> >     trend = Trend(
> >         phrase=query_phrase,
> >         content=json_trend_content,
> >         from_time=time_from,
> >         to_time=time_to,
> >         granularity=time_scale,
> >     )
> >     if trend is not None:
> >         try:
> >             db_trend = Trend.objects.get(pk=query_phrase)
> >             db_trend.delete()
> >             logger.info("delete old trend: %s. " % trend)
> >         except Trend.DoesNotExist:
> >             logger.info("create trend: %s. " % trend)
> >         trend.save()
> >     return trend_dict
> >
> > Thank you in advance!
> >
> > Roman
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> 
> 
> It looks like you can probably get rid of the try/except block at the end
> of the calculate_trend
> method as any existing Trend object will have already been caught in the
> first try/except block.
> 
> >From what I'm reading here:
> http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
> ,
> this issue can be caused by the client closing the connection before
> sendall() finishes writing. Can you estimate
> how long it takes for a request to this endpoint takes to resolve? If it's
> a long time(maybe due to the pandas call?),
> your browser/client may be timing out. It could be because it takes a while
> for the Db to find an existing Trend object
> as well.
> 
> I can't give you any advice on how to fix it exactly, but I can tell you
> what the problem is: The client is closing the
> connection before socket.sendall() has finished writing to the socket. My
> guess is that the calculate_trend() method
> takes a long time to complete and the client is timing out.

Thanks Justin,

I believe, the whole database story has no influence on the broken pipe error. I've commented out the whole block and leave only return line:
return HttpResponse(res, content_type="text/plain; charset=utf-8") 
The error is still present. And I have no influence on that.

I call python from js client:

    var newTrendReport = new App.TrendReport();
    newTrendReport.set('search_phrase', search_phrase);
    newTrendReport.set('time_from', time_from);
    newTrendReport.set('time_to', time_to);
    newTrendReport.set('time_scale', time_scale);
    newTrendReport.set('category', category);
    newTrendReport.startExport(

        function(response){
            console.log("Successfully calculated trend report.");
    		App.trendPage = new App.TrendPageView();
    		App.trendPage.render(response);
        },
    );

go throw:

    App.TrendReport = Backbone.Model.extend({
        urlRoot: "/api/trend_reports/",
        defaults: {
            search_phrase: "",
            time_from: "",
            time_to: "",
            time_scale: "",
            category: ""
        },

        startExportSuffix: "/export_report/",

        startExport: function( successCallback, errorCallback ) {
            console.log("start trend calculation");
            var that = this;
            var ajaxUrl = this.startExportSuffix;
            var options = {
                method: "POST",
                data: this.attributes,
                contentType: "application/json;charset=UTF-8",
                dataType: "json",

                error: errorCallback,
                success: successCallback
            };
            console.log("start trend export sync");
            App.ajax(ajaxUrl, options);
        }

    });

and come in export_report method.

My urls.py:

    url(r'^export_report', ensure_csrf_cookie(views.export_report), name="export_report"),


From rosuav at gmail.com  Mon Dec 12 11:17:25 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 13 Dec 2016 03:17:25 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <3924532.8F6SAcFxjW@PointedEars.de>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
Message-ID: <CAPTjJmodcg4X2VXZnsoGk4jwy+v6Q4CU4g5-RtpmkayetQcnmA@mail.gmail.com>

On Tue, Dec 13, 2016 at 1:34 AM, Thomas 'PointedEars' Lahn
<PointedEars at web.de> wrote:
> ?  How can one tell the difference in Python between a pre-initialized,
>    inherited attribute value and one own that is just equal to the inherited
>    one?  In ECMAScript, this.hasOwnProperty("foo") would return ?false? if
>    the property were inherited, ?true? otherwise.  But in Python,
>    hasattr(self, "foo") returns ?True? regardless whether the ?foo?
>    attribute of the calling instance has been assigned a value explicitly.
>    What is the Python equivalent of ECMAScript?s
>    Object.prototype.hasOwnProperty() method?

You could check "foo" in self.__dict__, but I don't know of any
real-world situations where you need to.

ChrisA


From juan0christian at gmail.com  Mon Dec 12 11:38:24 2016
From: juan0christian at gmail.com (Juan C.)
Date: Mon, 12 Dec 2016 14:38:24 -0200
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <584dfe8e$0$1602$c3e8da3$5496439d@news.astraweb.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <584dfe8e$0$1602$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAAp0bGthUy8ufu5thz_8EhMGKoiJWdApv92BNeDQ0wLEm9WbMA@mail.gmail.com>

On Sun, Dec 11, 2016 at 11:34 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:

> So... in summary:
>
>
> When *assigning* to an attribute:
>
> - use `self.attribute = ...` when you want an instance attribute;
>
> - use `Class.attribute = ...` when you want a class attribute in
>   the same class regardless of which subclass is being used;
>
> - use `type(self).attribute = ...` when you want a class attribute
>   in a subclass-friendly way.
>
>
> When *retrieving* an attribute:
>
> - use `self.attribute` when you want to use the normal inheritance
>   rules are get the instance attribute if it exists, otherwise a
>   class or superclass attribute;
>
> - use `type(self).attribute` when you want to skip the instance
>   and always return the class or superclass attribute.

Thanks, that seems simple enough.


From walters.justin01 at gmail.com  Mon Dec 12 12:38:25 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Mon, 12 Dec 2016 09:38:25 -0800
Subject: Django broken pipe error
In-Reply-To: <e246f8df-bead-4ae4-b41e-f18f2c1ba2bb@googlegroups.com>
References: <1030ce59-d2cb-4d90-b826-30637004f591@googlegroups.com>
 <843bd5a8-f127-4fdc-8817-0b1645c1872d@googlegroups.com>
 <CAO1D73H21KK14yheCLdH+LjXH_YWMaOoDM5Q7qDAm8kq1cS9QA@mail.gmail.com>
 <mailman.132.1481130930.2253.python-list@python.org>
 <e246f8df-bead-4ae4-b41e-f18f2c1ba2bb@googlegroups.com>
Message-ID: <CAO1D73F+SMyA=GNs4pNtO+q6n9SXk5ppMgU2GxbM5is-s6GsFw@mail.gmail.com>

On Mon, Dec 12, 2016 at 7:27 AM, roma <dr.roman.graf at gmail.com> wrote:

> Thanks Justin,
>
> I believe, the whole database story has no influence on the broken pipe
> error. I've commented out the whole block and leave only return line:
> return HttpResponse(res, content_type="text/plain; charset=utf-8")
> The error is still present. And I have no influence on that.
>
> I call python from js client:
>
>     var newTrendReport = new App.TrendReport();
>     newTrendReport.set('search_phrase', search_phrase);
>     newTrendReport.set('time_from', time_from);
>     newTrendReport.set('time_to', time_to);
>     newTrendReport.set('time_scale', time_scale);
>     newTrendReport.set('category', category);
>     newTrendReport.startExport(
>
>         function(response){
>             console.log("Successfully calculated trend report.");
>                 App.trendPage = new App.TrendPageView();
>                 App.trendPage.render(response);
>         },
>     );
>
> go throw:
>
>     App.TrendReport = Backbone.Model.extend({
>         urlRoot: "/api/trend_reports/",
>         defaults: {
>             search_phrase: "",
>             time_from: "",
>             time_to: "",
>             time_scale: "",
>             category: ""
>         },
>
>         startExportSuffix: "/export_report/",
>
>         startExport: function( successCallback, errorCallback ) {
>             console.log("start trend calculation");
>             var that = this;
>             var ajaxUrl = this.startExportSuffix;
>             var options = {
>                 method: "POST",
>                 data: this.attributes,
>                 contentType: "application/json;charset=UTF-8",
>                 dataType: "json",
>
>                 error: errorCallback,
>                 success: successCallback
>             };
>             console.log("start trend export sync");
>             App.ajax(ajaxUrl, options);
>         }
>
>     });
>
> and come in export_report method.
>
> My urls.py:
>
>     url(r'^export_report', ensure_csrf_cookie(views.export_report),
> name="export_report"),
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I'm not super familiar with the way backbone does http requests, but
something seems off about the startExport function.
It seems to me that you are sending a post request to "/export_report/"
which is an endpoint that I'm guessing is nested
in an include from "/api/trend_reports/". However, it looks like the error
you're getting above says you aren't sending
the request to "https://root.com/api/trend_reports/export_report/".
Instead, you are sending the request to "https://root.com/export_report/" .
Though, I'm also not sure that's the case because that would normally throw
a 404.

I also noticed that you set content type to 'application/json' in your js,
but the view function returns a 'text/plain' content type. Is
There a reason for this?

The data key in your js http function is set to the attributes variable
which, as far as I can tell, does not exist.

There's a lot going on here, but I think you can probably narrow it down to
something in your backbone code or the way
backbone handles http requests as the error you are getting is caused by
the client prematurely closing the socket.
It's possible backbone will stop reading the response since it isn't the
same content-type as the request.


From juan0christian at gmail.com  Mon Dec 12 12:57:28 2016
From: juan0christian at gmail.com (Juan C.)
Date: Mon, 12 Dec 2016 15:57:28 -0200
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <3924532.8F6SAcFxjW@PointedEars.de>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
Message-ID: <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>

On Mon, Dec 12, 2016 at 12:34 PM, Thomas 'PointedEars' Lahn
<PointedEars at web.de> wrote:
> First of all, the proper term for what you are doing there is _not_ ?call?;
> you are _accessing_ an attribute instead.

Indeed, `accessing` seems better. I was looking for a better word but
couldn't find at the moment.

> To call something means generally in programming, and in Python, to execute
> it as a function instead: In the code above, the class object referred to by
> ?Box? is called twice in order to instantiate twice (calling a class object
> in Python means to instantiate it, implicitly calling its constructor
> method, ?__init__?, if any), and the global ?print? function is called three
> times.

Since we are talking about Python terminology I believe that calling
`__init__` a constructor is also wrong. I've already seem some
discussions regarding it and the general consensus is that `__init__`
shouldn't be called constructor as it isn't really a constructor (like
Java/C# constructors). Some source:

- http://stackoverflow.com/questions/4859129/python-and-python-c-api-new-versus-init
- http://stackoverflow.com/questions/674304/pythons-use-of-new-and-init


From gordon at panix.com  Mon Dec 12 13:34:36 2016
From: gordon at panix.com (John Gordon)
Date: Mon, 12 Dec 2016 18:34:36 +0000 (UTC)
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
Message-ID: <o2mqjr$k9c$1@reader1.panix.com>

In <mailman.16.1481487105.2322.python-list at python.org> "Juan C." <juan0christian at gmail.com> writes:

> The instructor said that the right way to call a class attribute is to use
> 'Class.class_attr' notation, but on the web I found examples where people
> used 'self.class_attr' to call class attributes.

Class instances may override class attributes by creating local attributes
of the same name, in which case Class.class_attr and self.class_attr may
not be equal, so you'd have to use the correct one for your needs.

If you're sure that an instance does not override the class attribute,
then you can use whichever one you prefer.  self.class_attr may be more
convenient because you don't have to provide a specific class name.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From saxri89 at gmail.com  Mon Dec 12 14:29:29 2016
From: saxri89 at gmail.com (Xristos Xristoou)
Date: Mon, 12 Dec 2016 11:29:29 -0800 (PST)
Subject: SIP install error on windows
Message-ID: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>


hello i want to install sip on windows bit using python 
 32 bit.
i download sip from this link https://www.riverbankcomputing.com
i try to install from cmd line :

C:\WINDOWS\system32>cd C:\Users\username\Desktop\sip-4.17

C:\Users\username\Desktop\sip-4.17>python configure.py install
and take that error any idea ?

This is SIP 4.17 for Python 2.7.11 on win32.
Error: Unsupported macro name specified. Use the --show-build-macros flag to
see a list of supported macros.



From phil at riverbankcomputing.com  Mon Dec 12 14:33:45 2016
From: phil at riverbankcomputing.com (Phil Thompson)
Date: Mon, 12 Dec 2016 19:33:45 +0000
Subject: SIP install error on windows
In-Reply-To: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
References: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
Message-ID: <E1EA5645-E689-4CA1-A466-C7FA7DBB14FB@riverbankcomputing.com>

On 12 Dec 2016, at 7:29 pm, Xristos Xristoou <saxri89 at gmail.com> wrote:
> 
> 
> hello i want to install sip on windows bit using python 
> 32 bit.
> i download sip from this link https://www.riverbankcomputing.com
> i try to install from cmd line :
> 
> C:\WINDOWS\system32>cd C:\Users\username\Desktop\sip-4.17
> 
> C:\Users\username\Desktop\sip-4.17>python configure.py install
> and take that error any idea ?
> 
> This is SIP 4.17 for Python 2.7.11 on win32.
> Error: Unsupported macro name specified. Use the --show-build-macros flag to
> see a list of supported macros.

Try the documentation...

http://pyqt.sourceforge.net/Docs/sip4/installation.html

Phil

From bgailer at gmail.com  Mon Dec 12 14:38:37 2016
From: bgailer at gmail.com (Bob Gailer)
Date: Mon, 12 Dec 2016 14:38:37 -0500
Subject: SIP install error on windows
In-Reply-To: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
References: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
Message-ID: <CAP1rxO4dDFpPoBoDX9xDqkRzeNRvWE9xDD0O22TfW1Jcu7Bj7w@mail.gmail.com>

On Dec 12, 2016 2:30 PM, "Xristos Xristoou" <saxri89 at gmail.com> wrote:
>
>
> hello i want to install sip on windows bit using python
>  32 bit.
> i download sip from this link https://www.riverbankcomputing.com
> i try to install from cmd line :
>
> C:\WINDOWS\system32>cd C:\Users\username\Desktop\sip-4.17
>
> C:\Users\username\Desktop\sip-4.17>python configure.py install
> and take that error any idea ?
The instructions at Riverbank for installing sip tell me to use pip3
install. I suggest you try that.
>
> This is SIP 4.17 for Python 2.7.11 on win32.
> Error: Unsupported macro name specified. Use the --show-build-macros flag
to
> see a list of supported macros.
>
> --
> https://mail.python.org/mailman/listinfo/python-list


From phil at riverbankcomputing.com  Mon Dec 12 14:40:53 2016
From: phil at riverbankcomputing.com (Phil Thompson)
Date: Mon, 12 Dec 2016 19:40:53 +0000
Subject: SIP install error on windows
In-Reply-To: <CAP1rxO4dDFpPoBoDX9xDqkRzeNRvWE9xDD0O22TfW1Jcu7Bj7w@mail.gmail.com>
References: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
 <CAP1rxO4dDFpPoBoDX9xDqkRzeNRvWE9xDD0O22TfW1Jcu7Bj7w@mail.gmail.com>
Message-ID: <16D561E4-1F23-4A35-8283-54A40827CA4A@riverbankcomputing.com>

On 12 Dec 2016, at 7:38 pm, Bob Gailer <bgailer at gmail.com> wrote:
> 
> On Dec 12, 2016 2:30 PM, "Xristos Xristoou" <saxri89 at gmail.com> wrote:
>> 
>> 
>> hello i want to install sip on windows bit using python
>> 32 bit.
>> i download sip from this link https://www.riverbankcomputing.com
>> i try to install from cmd line :
>> 
>> C:\WINDOWS\system32>cd C:\Users\username\Desktop\sip-4.17
>> 
>> C:\Users\username\Desktop\sip-4.17>python configure.py install
>> and take that error any idea ?
> The instructions at Riverbank for installing sip tell me to use pip3
> install. I suggest you try that.

Not for Python v2.

Phil


From saxri89 at gmail.com  Mon Dec 12 14:44:40 2016
From: saxri89 at gmail.com (Xristos Xristoou)
Date: Mon, 12 Dec 2016 11:44:40 -0800 (PST)
Subject: SIP install error on windows
In-Reply-To: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
References: <b96a66ba-c9a7-4eb2-9ff5-de58ce8a0736@googlegroups.com>
Message-ID: <1df292b3-c6d1-4552-b420-4c4eb43a74bf@googlegroups.com>

?? ???????, 12 ?????????? 2016 - 9:29:38 ?.?. UTC+2, ? ??????? Xristos Xristoou ??????:
> hello i want to install sip on windows bit using python 
>  32 bit.
> i download sip from this link https://www.riverbankcomputing.com
> i try to install from cmd line :
> 
> C:\WINDOWS\system32>cd C:\Users\username\Desktop\sip-4.17
> 
> C:\Users\username\Desktop\sip-4.17>python configure.py install
> and take that error any idea ?
> 
> This is SIP 4.17 for Python 2.7.11 on win32.
> Error: Unsupported macro name specified. Use the --show-build-macros flag to
> see a list of supported macros.

yes but i dont have python 3 i need use python 2.7


From ned at nedbatchelder.com  Mon Dec 12 15:15:13 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 12 Dec 2016 12:15:13 -0800 (PST)
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <mailman.27.1481565498.2322.python-list@python.org>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
Message-ID: <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>

On Monday, December 12, 2016 at 12:58:30 PM UTC-5, Juan C. wrote:
> Since we are talking about Python terminology I believe that calling
> `__init__` a constructor is also wrong. I've already seem some
> discussions regarding it and the general consensus is that `__init__`
> shouldn't be called constructor as it isn't really a constructor (like
> Java/C# constructors). Some source:
> 
> - http://stackoverflow.com/questions/4859129/python-and-python-c-api-new-versus-init
> - http://stackoverflow.com/questions/674304/pythons-use-of-new-and-init

Claiming that __init__ isn't a constructor seems overly pedantic to me.
What's true is that Python's constructors (__init__) are different than
C++ constructors.  In C++, you don't have an object of type T until the
constructor has finished. In Python, you have an object of type T before
__init__ has been entered.

The reason to call __init__ a constructor is because of what is the same
between C++ and Python: the constructor is where the author of the class
can initialize instances of the class.

There are many programming languages, and they have similar overlapping
constructs, but there are differences among the constructs.  It's a
challenge to decide when those differences are great enough to use a
different name, and when those differences are slight enough to just note
them as a difference.

As an example, people are happy to use the word "function" for things in
Python, C, C++, Java, JavaScript, and Haskell, despite the huge differences
in how they behave.  The commonalities, especially in how these constructs
are used by developers to express themselves, are similar enough that
we call them all "functions."  It seems to me that we can do the same
for "constructor."

--Ned.


From PointedEars at web.de  Mon Dec 12 15:59:40 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Mon, 12 Dec 2016 21:59:40 +0100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
Message-ID: <1782884.oMNUckLgyt@PointedEars.de>

Juan C. wrote:

> On Mon, Dec 12, 2016 at 12:34 PM, Thomas 'PointedEars' Lahn
> <PointedEars at web.de> wrote:
>> To call something means generally in programming, and in Python, to
>> execute it as a function instead: In the code above, the class object
>> referred to by ?Box? is called twice in order to instantiate twice
>> (calling a class object in Python means to instantiate it, implicitly
>> calling its constructor method, ?__init__?, if any), and the global
>> ?print? function is called three times.
> 
> Since we are talking about Python terminology I believe that calling
> `__init__` a constructor is also wrong. I've already seem some
                                   ^^^^^
> discussions regarding it and the general consensus is that `__init__`
> shouldn't be called constructor as it isn't really a constructor (like
> Java/C# constructors). [?]

IBTD:

,-<http://foldoc.org/constructor>
| 
| constructor
| 
| (programming)   1. In object-oriented languages, a function provided by a 
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^
| class to initialise a newly created object. The constructor function 
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| typically has the same name as the class. It may take arguments, e.g. to 
| set various attributes of the object or it may just leave everything 
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| undefined to be set elsewhere.
| 
| A class may also have a destructor function that is called when objects of 
| the class are destroyed.

Free On-line Dictionary of Computing, Last updated: 2014-10-04.

In Python, a class?s __init__() is called after its __new__() [1], and 
?typically? means ?not always? (for another example, in clean PHP code the 
constructor?s name is ?__construct?).

<https://docs.python.org/3/reference/datamodel.html?highlight=__init__#object.__init__>

[In fact, there are constructors in object-oriented programming languages 
with prototype-based inheritance, like implementations of ECMAScript Ed. 1 
to 7 (except 4), too.]

-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From greg.ewing at canterbury.ac.nz  Mon Dec 12 16:30:47 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 13 Dec 2016 10:30:47 +1300
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
Message-ID: <eb8j88Fco3aU1@mid.individual.net>

Ned Batchelder wrote:
> In C++, you don't have an object of type T until the
> constructor has finished. In Python, you have an object of type T before
> __init__ has been entered.

That distinction seems a bit pedantic as well. Inside a C++
constructor you have access to something having all the
fields and methods of an object of type T, they just
haven't been filled in yet.

It's a bit like asking at what point between conception and
birth a baby starts to exist.

-- 
Greg



From steve+python at pearwood.info  Mon Dec 12 16:50:54 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 13 Dec 2016 08:50:54 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAPTjJmodcg4X2VXZnsoGk4jwy+v6Q4CU4g5-RtpmkayetQcnmA@mail.gmail.com>
 <mailman.24.1481559448.2322.python-list@python.org>
Message-ID: <584f1bbf$0$1586$c3e8da3$5496439d@news.astraweb.com>

On Tue, 13 Dec 2016 03:17 am, Chris Angelico wrote:

> You could check "foo" in self.__dict__, but I don't know of any
> real-world situations where you need to.

vars(self) is probably the better way to access self's namespace, rather
than directly self.__dict__. Unfortunately vars() doesn't understand
__slots__, which I think is an oversight.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ned at nedbatchelder.com  Mon Dec 12 16:55:08 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 12 Dec 2016 13:55:08 -0800 (PST)
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <eb8j88Fco3aU1@mid.individual.net>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <eb8j88Fco3aU1@mid.individual.net>
Message-ID: <5e175c03-4afe-4c46-a3fe-86dd3ef6f583@googlegroups.com>

On Monday, December 12, 2016 at 4:31:00 PM UTC-5, Gregory Ewing wrote:
> Ned Batchelder wrote:
> > In C++, you don't have an object of type T until the
> > constructor has finished. In Python, you have an object of type T before
> > __init__ has been entered.
> 
> That distinction seems a bit pedantic as well. Inside a C++
> constructor you have access to something having all the
> fields and methods of an object of type T, they just
> haven't been filled in yet.

I agree the distinction is pedantic. It does matter sometimes, such as:
if a C++ constructor raises an exception, will the corresponding destructor
be run, or not? (No, because it never finished making an object of type T.)

But in any case, all the more reason not to let C++ semantics govern
Python vocabulary.

--Ned.


From juan0christian at gmail.com  Mon Dec 12 17:08:42 2016
From: juan0christian at gmail.com (Juan C.)
Date: Mon, 12 Dec 2016 20:08:42 -0200
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <1782884.oMNUckLgyt@PointedEars.de>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <1782884.oMNUckLgyt@PointedEars.de>
Message-ID: <CAAp0bGuquP9PyK+XOD6-N4-_sM8UoBt=63Nc_zY5e-xt6ctW-Q@mail.gmail.com>

On Mon, Dec 12, 2016 at 6:59 PM, Thomas 'PointedEars' Lahn
<PointedEars at web.de> wrote:
> <https://docs.python.org/3/reference/datamodel.html?highlight=__init__#object.__init__>

Using the Python official doc link you provided, it clearly states
that `__new__` is the one called to "create a new instance of class
[...] The return value of __new__() should be the new object instance
(usually an instance of cls)." On the other hand, `__init__` is
"called after the instance has been created (by __new__()), but before
it is returned to the caller."

Here we have the same mindset regarding `__new__` vs `__init__`:

- http://python-textbok.readthedocs.io/en/1.0/Classes.html
"Note: __init__ is sometimes called the object?s constructor, because
it is used similarly to the way that constructors are used in other
languages, but that is not technically correct ? it?s better to call
it the initialiser. There is a different method called __new__ which
is more analogous to a constructor, but it is hardly ever used."

- http://www.python-course.eu/python3_object_oriented_programming.php
"We want to define the attributes of an instance right after its
creation. __init__ is a method which is immediately and automatically
called after an instance has been created. [...] The __init__ method
is used to initialize an instance."

- https://en.wikipedia.org/wiki/Constructor_(object-oriented_programming)#Python
"In Python, constructors are defined by one or both of __new__ and
__init__ methods. A new instance is created by calling the class as if
it were a function, which calls the __new__ and __init__ methods. If a
constructor method is not defined in the class, the next one found in
the class's Method Resolution Order will be called."

- http://www.diveintopython3.net/iterators.html
"The __init__() method is called immediately after an instance of the
class is created. It would be tempting ? but technically incorrect ?
to call this the ?constructor? of the class. It?s tempting, because it
looks like a C++ constructor (by convention, the __init__() method is
the first method defined for the class), acts like one (it?s the first
piece of code executed in a newly created instance of the class), and
even sounds like one. Incorrect, because the object has already been
constructed by the time the __init__() method is called, and you
already have a valid reference to the new instance of the class."


In general, the idea is simple, `__new__` constructs and `__init__`
initializes, this is what I believe in, after all the name `__init__`
already tell us that it's a *init* ialiser...


From steve+python at pearwood.info  Mon Dec 12 17:16:12 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 13 Dec 2016 09:16:12 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
Message-ID: <584f21ad$0$1618$c3e8da3$5496439d@news.astraweb.com>

On Tue, 13 Dec 2016 07:15 am, Ned Batchelder wrote:

> Claiming that __init__ isn't a constructor seems overly pedantic to me.
> What's true is that Python's constructors (__init__) are different than
> C++ constructors.  In C++, you don't have an object of type T until the
> constructor has finished. In Python, you have an object of type T before
> __init__ has been entered.
> 
> The reason to call __init__ a constructor is because of what is the same
> between C++ and Python: the constructor is where the author of the class
> can initialize instances of the class.

That logic would have been more convincing back in the days of classic
classes in Python 1.5 or 2.0, less so after class/type unification where we
have an actual constructor __new__ that creates the instance.

The section of the docs that deal with object customization takes an
agnostic position, referring only to the "class constructor EXPRESSION"
(i.e. MyClass(spam, eggs) or equivalent):

https://docs.python.org/2/reference/datamodel.html#object.__init__

and similarly in __new__ (this time the v3 docs, just because):

https://docs.python.org/3/reference/datamodel.html#object.__new__


so if you want to be pedantic, one might argue that Python has a concept
of "object/class constructor expressions", MyClass(spam, eggs), but not of
constructor method(s). But that seems even stranger than insisting that:

- __new__ is the constructor method;
- __init__ is the initializer method.


Another way of naming things is to say that:

- __new__ is the allocator;
- __init__ is the initializer;
- the two together, __new__ + __init__, make up the constructor.

Not to be confused with the constructor expression, MyClass(spam, eggs), or
the alternative constructor, MyClass.make_instance(...).

Naming things is hard.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ben+python at benfinney.id.au  Mon Dec 12 18:17:03 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 13 Dec 2016 10:17:03 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
Message-ID: <8560mo20wg.fsf@benfinney.id.au>

Ned Batchelder <ned at nedbatchelder.com> writes:

> Claiming that __init__ isn't a constructor seems overly pedantic to
> me.

Whereas to me, claiming that ?Foo.__init__? is a constructor seems
needlessly confusing.

* Classes already have a constructor, ?Foo.__new__?. If we call
  something else the constructor, what do we call ?__new__?? Are they
  both constructors?

* A constructor for Foo should start from ?no instance? and result in
  ?an instance of Foo?. ?Foo.__init__? does not do that; ?Foo.__new__?
  does.

* A constructor should return the instance. ?Foo.__init__? does not do
  that; ?Foo.__new__? does.

If the differences didn't matter I would agree that ?overly pedantic? is
fair. But those differences trip up newcomers. Thinking of
?Foo.__init__? leads people to wonder where the ?self? attribute came
from ? am I not meant to be constructing it? ? and to attempt to return
that instance. And when the time comes to lean about ?__new__? the
confusion continues, because the newcomer has been told that something
*else* is the constructor, so what's this?

> What's true is that Python's constructors (__init__) are different than
> C++ constructors.  In C++, you don't have an object of type T until the
> constructor has finished. In Python, you have an object of type T before
> __init__ has been entered.

I'm not going to argue that C++ should define terminology for other
languages. But ?constructor? should have a close correlation with the
normal English-language meaning of the term.

That meaning matches ?Foo.__new__?, which makes that method a
constructor. It does not match ?Foo.__init__?, which makes that method
not a constructor.

> The reason to call __init__ a constructor is because of what is the
> same between C++ and Python: the constructor is where the author of
> the class can initialize instances of the class.

So you've just described what ?Foo._init__? does: it initialises the
existing instance. That's why it is better to call it the ?initialiser?,
a term we already have and use correctly.

-- 
 \     ?DRM doesn't inconvenience [lawbreakers] ? indeed, over time it |
  `\     trains law-abiding users to become [lawbreakers] out of sheer |
_o__)                        frustration.? ?Charles Stross, 2010-05-09 |
Ben Finney



From rosuav at gmail.com  Mon Dec 12 18:23:57 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 13 Dec 2016 10:23:57 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <8560mo20wg.fsf@benfinney.id.au>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
Message-ID: <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>

On Tue, Dec 13, 2016 at 10:17 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> If the differences didn't matter I would agree that ?overly pedantic? is
> fair. But those differences trip up newcomers. Thinking of
> ?Foo.__init__? leads people to wonder where the ?self? attribute came
> from ? am I not meant to be constructing it? ? and to attempt to return
> that instance. And when the time comes to lean about ?__new__? the
> confusion continues, because the newcomer has been told that something
> *else* is the constructor, so what's this?

In JavaScript, it's normal to talk about "calling a function as a
constructor". When you do, there is a 'this' object before you start.
Should we tell the JavaScript folks to be more pedantic, because
'this' should end up existing? Does it really even matter when memory
gets allocated and the object's identity assigned? Before __init__
gets called, the object isn't "truly there" - its fundamental
invariants may not yet have been established, and key attributes might
not have been set up. Once __init__ finishes, there is an expectation
that attributes and invariants are sorted out.

This is like the old "Python doesn't have variables" thing.
Ultimately, every language has slightly different semantics (otherwise
they'd be trivial transformations, like Ook and Brainf*), so you have
to learn that the "constructor" might have slightly different
semantics. Accept it - embrace it. Learn it.

ChrisA


From ben+python at benfinney.id.au  Mon Dec 12 18:57:09 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 13 Dec 2016 10:57:09 +1100
Subject: Python constructors have particular semantics, and
 =?utf-8?Q?=E2=80=98Foo=2E=5F=5Finit=5F=5F=E2=80=99?= doesn't qualify (was:
 The right way to 'call' a class attribute inside the same class)
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
Message-ID: <851sxc1z1m.fsf_-_@benfinney.id.au>

Chris Angelico <rosuav at gmail.com> writes:

> On Tue, Dec 13, 2016 at 10:17 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > If the differences didn't matter I would agree that ?overly
> > pedantic? is fair. But those differences trip up newcomers. Thinking
> > of ?Foo.__init__? leads people to wonder where the ?self? attribute
> > came from ? am I not meant to be constructing it? ? and to attempt
> > to return that instance. And when the time comes to lean about
> > ?__new__? the confusion continues, because the newcomer has been
> > told that something *else* is the constructor, so what's this?
>
> In JavaScript [different semantics apply].
>
> Ultimately, every language has slightly different semantics [?], so
> you have to learn that the "constructor" might have slightly different
> semantics.

Please read again the message you extracted that quote from. I already
said I'm not claiming some other programming language's semantics should
dictate Python's.


What I'm saying is that in Python, there *already are* different
semantics for a constructor, and they don't match the semantics of
?Foo.__init__?.

In Python, a constructor for a class is a class method. ?Foo.__new__? is
a constructor. ?Foo.__init__? is an instance method, so it's not a
constructor.

In Python, a constructor for a class makes the instance where it didn't
already exist. ?Foo.__new__? is a constructor. ?Foo.__init__? requires
the instance to already be constructed, so it's not a constructor.

In Python, a constructor for a class returns a new instance of that
class. ?Foo.__new__? is a constructor. ?Foo.__init__? must return None,
so it's not a constructor.

In Python, a custom constructor for a class follows the above
descriptions. ?datetime.fromtimestamp? is a constructor.
?datetime.__init__? is not.


None of this argues from the semantics of other programming languages,
so telling me other languages have different semantics is not a response
to this argument.

I'm showing that Python classes *already have* constructors, and
?Foo.__init__? doesn't qualify because it doesn't have the semantics of
Python constructors.

-- 
 \     ?I've always wanted to be somebody, but I see now that I should |
  `\           have been more specific.? ?Jane Wagner, via Lily Tomlin |
_o__)                                                                  |
Ben Finney



From juan0christian at gmail.com  Mon Dec 12 20:07:50 2016
From: juan0christian at gmail.com (Juan C.)
Date: Mon, 12 Dec 2016 23:07:50 -0200
Subject: =?UTF-8?Q?Re=3A_Python_constructors_have_particular_semantics=2C_a?=
 =?UTF-8?Q?nd_=E2=80=98Foo=2E=5F=5Finit=5F=5F=E2=80=99_doesn=27t_qualify_=28was=3A_The_right_way_to?=
 =?UTF-8?Q?_=27call=27_a_class_attribute_inside_the_same_class=29?=
In-Reply-To: <851sxc1z1m.fsf_-_@benfinney.id.au>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
Message-ID: <CAAp0bGtiUxkbDkBqwrsZJBCzrTmBeB=wZrXRV9AqsUnL-t3tTQ@mail.gmail.com>

I agree with you, I'll post here the same thing I said in there for
another member:

On Mon, Dec 12, 2016 at 6:59 PM, Thomas 'PointedEars' Lahn
<PointedEars at web.de> wrote:
> <https://docs.python.org/3/reference/datamodel.html?highlight=__init__#object.__init__>
>

Using the Python official doc link you provided, it clearly states
that `__new__` is the one called to "create a new instance of class
[...] The return value of __new__() should be the new object instance
(usually an instance of cls)."

On the other hand, `__init__` is "called after the instance has been
created (by __new__()), but before it is returned to the caller."

Here we have the same mindset regarding `__new__` vs `__init__`:

- http://python-textbok.readthedocs.io/en/1.0/Classes.html
"Note: __init__ is sometimes called the object?s constructor, because
it is used similarly to the way that constructors are used in other
languages, but that is not technically correct ? it?s better to call
it the initialiser. There is a different method called __new__ which
is more analogous to a constructor, but it is hardly ever used."

- http://www.python-course.eu/python3_object_oriented_programming.php
"We want to define the attributes of an instance right after its
creation. __init__ is a method which is immediately and automatically
called after an instance has been created. [...] The __init__ method
is used to initialize an instance."

- https://en.wikipedia.org/wiki/Constructor_(object-oriented_programming)#Python
"In Python, constructors are defined by one or both of __new__ and
__init__ methods. A new instance is created by calling the class as if
it were a function, which calls the __new__ and __init__ methods. If a
constructor method is not defined in the class, the next one found in
the class's Method Resolution Order will be called."

- http://www.diveintopython3.net/iterators.html
"The __init__() method is called immediately after an instance of the
class is created. It would be tempting ? but technically incorrect ?
to call this the ?constructor? of the class. It?s tempting, because it
looks like a C++ constructor (by convention, the __init__() method is
the first method defined for the class), acts like one (it?s the first
piece of code executed in a newly created instance of the class), and
even sounds like one. Incorrect, because the object has already been
constructed by the time the __init__() method is called, and you
already have a valid reference to the new instance of the class."

In general, the idea is simple, `__new__` constructs and `__init__`
initializes, this is what I believe in, after all the name `__init__`
already tell us that it's a *init* ialiser...

It doesn't matter if Java, C#, Javascript, <you name it> have
different approaches, I'm programming (or at least, trying to :p) in
Python, so I'll follow what the official doc and the vast majority of
books/courses/etc say.


From ned at nedbatchelder.com  Mon Dec 12 20:12:32 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 12 Dec 2016 17:12:32 -0800 (PST)
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <mailman.36.1481584642.2322.python-list@python.org>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <mailman.36.1481584642.2322.python-list@python.org>
Message-ID: <685aad0c-d852-4601-9cdd-d5b9ec05a833@googlegroups.com>

On Monday, December 12, 2016 at 6:17:43 PM UTC-5, Ben Finney wrote:
> Ned Batchelder <ned at nedbatchelder.com> writes:
> 
> > Claiming that __init__ isn't a constructor seems overly pedantic to
> > me.
> 
> Whereas to me, claiming that ?Foo.__init__? is a constructor seems
> needlessly confusing.
> 
...
> * A constructor should return the instance. ?Foo.__init__? does not do
>   that; ?Foo.__new__? does.

This seems like an odd thing to insist on. C++ and Java constructors
don't return the instance.  C++ allocators return the instance. 
I'm happy to call __new__ an allocator. It serves exactly the same
role in Python as allocators do in C++: its job is to create the raw
material for the object, and they very rarely need to be written.

> 
> If the differences didn't matter I would agree that ?overly pedantic? is
> fair. But those differences trip up newcomers. Thinking of
> ?Foo.__init__? leads people to wonder where the ?self? attribute came
> from ? am I not meant to be constructing it? ? and to attempt to return
> that instance.

Creating objects is confusing, but beginners don't come to Python
with an expectation of what a "constructor" is.  Filling in the 
attributes of an object is just as clearly a kind of construction
as allocating memory.

--Ned.


From bussonniermatthias at gmail.com  Mon Dec 12 23:57:34 2016
From: bussonniermatthias at gmail.com (Matthias Bussonnier)
Date: Mon, 12 Dec 2016 20:57:34 -0800 (PST)
Subject: Method to know if object support being weakreferenced ?
Message-ID: <f1f4b779-bb8b-4b57-9413-06861dff121c@googlegroups.com>

Hi all, 

I was recently had to use weakreferences, using the weakref module, and came across the fact that some object cannot be weakreferenced. If you try to do so you get greated by a TypeError, which is a totally expected and documented behavior. 

As I tend to prefer the "Look before you leap" approach I search for a method capable of telling me whether an object can be weakreferenced. Which I failed to found. 

I could of course write a function that try/except and return False/True depending on the result, but that seem suboptimal as how can I know that the TypeError does come from not being able to take a weak reference ? And not from something else ?

The common Idiom in CPython, at the C layer seem to be PyType_SUPPORTS_WEAKREFS(Py_TYPE(ob)). So I can (and did) write a C-extension that expose such a function, but it does seem silly that no one else did that before me, and that no one seemed to have encountered the problem before. 

So am I missing something ? Is such a function not useful ?  Is there any reason not to have it in the stdlib ?

Thanks.
-- 
M


From steve+comp.lang.python at pearwood.info  Tue Dec 13 00:36:31 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 13 Dec 2016 16:36:31 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <mailman.36.1481584642.2322.python-list@python.org>
 <685aad0c-d852-4601-9cdd-d5b9ec05a833@googlegroups.com>
Message-ID: <584f88e2$0$2862$c3e8da3$76491128@news.astraweb.com>

On Tuesday 13 December 2016 12:12, Ned Batchelder wrote:

> On Monday, December 12, 2016 at 6:17:43 PM UTC-5, Ben Finney wrote:
>> Ned Batchelder <ned at nedbatchelder.com> writes:
>> 
>> > Claiming that __init__ isn't a constructor seems overly pedantic to
>> > me.
>> 
>> Whereas to me, claiming that ?Foo.__init__? is a constructor seems
>> needlessly confusing.
>> 
> ...
>> * A constructor should return the instance. ?Foo.__init__? does not do
>>   that; ?Foo.__new__? does.
> 
> This seems like an odd thing to insist on. C++ and Java constructors
> don't return the instance.

I'm not sure what you mean by that. I think they do, at least in C++.

Normally in C++ you would create an instance like this:

MyClass obj(arg);


In Python terms, that would be written as: `obj = MyClass(arg)`.


But one can also create "anonymous objects" which aren't assigned to a named 
variable:

something.method( MyClass(args) );


Doesn't that count as "returning an instance"?


> C++ allocators return the instance.

As I understand it, C++ allocators are specialist methods used in the standard 
library for container classes which allocate storage, i.e. the equivalent of 
Python lists and dicts.

http://en.cppreference.com/w/cpp/concept/Allocator

As far as I can tell, they aren't relevant to "simple" record- or struct-like 
objects.


> I'm happy to call __new__ an allocator. It serves exactly the same
> role in Python as allocators do in C++: its job is to create the raw
> material for the object, and they very rarely need to be written.

I think that last part is wrong in Python. Any time you are creating an 
immutable class -- and some of us do that a lot -- you generally need to write 
__new__. That makes __new__ very different from C++ allocators, and more like 
C++ constructors.

On the other hand, __new__ seems to be quite similar to ObjectiveC's `alloc`, 
which is called the allocator, and __init__ similar to ObjectiveC's init, which 
is the initialiser!

ObjectiveC also has a new method, which just calls alloc then init.


>> If the differences didn't matter I would agree that ?overly pedantic? is
>> fair. But those differences trip up newcomers. Thinking of
>> ?Foo.__init__? leads people to wonder where the ?self? attribute came
>> from ? am I not meant to be constructing it? ? and to attempt to return
>> that instance.
> 
> Creating objects is confusing, but beginners don't come to Python
> with an expectation of what a "constructor" is.  Filling in the
> attributes of an object is just as clearly a kind of construction
> as allocating memory.


Remember that in Python __new__ can do a lot more than just allocate memory.

It just seems really weird to describe __init__ as a constructor when it 
initialises an existing instance: "initialiser" just seems to be the obvious 
term for it. (Do I really need to spell it out? __INIT__/INITialiser.)

Python's object creation model seems to be a lot closer to that of ObjectiveC 
or C++ than Javascript, so copying Javascript's terminology seems risky to me.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Tue Dec 13 00:45:41 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 13 Dec 2016 16:45:41 +1100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <mailman.37.1481585045.2322.python-list@python.org>
Message-ID: <584f8b08$0$1585$c3e8da3$5496439d@news.astraweb.com>

On Tuesday 13 December 2016 10:23, Chris Angelico wrote:

> On Tue, Dec 13, 2016 at 10:17 AM, Ben Finney <ben+python at benfinney.id.au>
> wrote:
>> If the differences didn't matter I would agree that ?overly pedantic? is
>> fair. But those differences trip up newcomers. Thinking of
>> ?Foo.__init__? leads people to wonder where the ?self? attribute came
>> from ? am I not meant to be constructing it? ? and to attempt to return
>> that instance. And when the time comes to lean about ?__new__? the
>> confusion continues, because the newcomer has been told that something
>> *else* is the constructor, so what's this?
> 
> In JavaScript, it's normal to talk about "calling a function as a
> constructor". When you do, there is a 'this' object before you start.
> Should we tell the JavaScript folks to be more pedantic, because
> 'this' should end up existing? 

You've ignored Ben's critical point that if we call __init__ the "constructor", 
what do we call __new__? (Perhaps we should call __new__ the "initialiser" for 
maximum confusion.)

I don't understand the point of bringing up Javascript. Ben has already said 
that we shouldn't feel the need to mindlessly copy C++ terminology. Is it your 
position that we *should* copy Javascript terminology? Why Javascript and not 
C++ or ObjectiveC? Even when it goes against the obvious English mnemonic?

    __init__ is the INITialiser (it initialises an existing instance);

    __new__ creates/constructs a NEW instance


> Does it really even matter when memory
> gets allocated and the object's identity assigned? 

Actually, yes it does. Try constructing an immutable object like a subclass of 
float, str or int from the __init__ method. Attaching attributes to the 
instance doesn't count.


class PositiveInt(int):
    def __init__(self, arg):
        arg = abs(arg)
        return super().__init__(self, arg)


So much fail...



> Before __init__
> gets called, the object isn't "truly there" - its fundamental
> invariants may not yet have been established, and key attributes might
> not have been set up.

"Fundamental invariants" is tricky though -- are they *truly* fundamental? This 
is Python -- I can easily make a Python-based class that lacks the attributes 
that its methods assume will be there, or delete them after creation.

But there is one thing which truly is fundamental: the instance creation, the 
moment that object.__new__ returns a new instance. Before calling that, the 
instance genuinely doesn't exist; after object.__new__ returns, it genuinely 
does. (Even if it isn't fully initialised.)

Unlike __init__, object.__new__ is atomic: it either succeeds, or it doesn't. 
So object.__new__ really is the constructor of instances, and so following 
standard practice, MyClass.__new__ which inherits from object ought to be 
called the same thing. (Even if it is no longer atomic, due to being written in 
Python.)


> Once __init__ finishes, there is an expectation
> that attributes and invariants are sorted out.
> 
> This is like the old "Python doesn't have variables" thing.
> Ultimately, every language has slightly different semantics (otherwise
> they'd be trivial transformations, like Ook and Brainf*), so you have
> to learn that the "constructor" might have slightly different
> semantics. Accept it - embrace it. Learn it.

We're not debating what other languages should call __new__ and __init__ or 
whatever their equivalents are. We're debating what Python should call them.

For some prior art, consider ObjectiveC. To create a new instance of a class, 
you call:

[[MyClass alloc] init]

where alloc is the "allocator" (it basically just allocates memory, and very 
little else) and init is the "initialiser" (because it initialises the newly 
created instance.

In more modern versions of ObjectiveC, there's a short-cut:

[MyClass new]

where new calls alloc then init for you.

In Ruby, people think of two distinct things as the "constructor", depending on 
what you are doing. [Source: my resident Ruby expert at work.] If you are 
talking about *writing* a class, the constructor is the initialize method:

class MyClass
    def initialize
        ...
    end
end

But if you are talking about *creating an instance* it is the new method:

MyClass.new

which itself automatically calls initialize behind the scenes. (I'm told that 
it is possible to override new, but nobody does it.)


It seems to me that both of these are quite similar to Python's model.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Tue Dec 13 01:00:02 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 13 Dec 2016 17:00:02 +1100
Subject: Method to know if object support being weakreferenced ?
References: <f1f4b779-bb8b-4b57-9413-06861dff121c@googlegroups.com>
Message-ID: <584f8e63$0$1503$c3e8da3$5496439d@news.astraweb.com>

On Tuesday 13 December 2016 15:57, Matthias Bussonnier wrote:

[...]
> I could of course write a function that try/except and return False/True
> depending on the result, but that seem suboptimal as how can I know that the
> TypeError does come from not being able to take a weak reference ? And not
> from something else ?

Do you mean something like this?

try:
    proxy = weakref.proxy(obj)
except TypeError:
    proxy = None


Where else could it come from?

> The common Idiom in CPython, at the C layer seem to be
> PyType_SUPPORTS_WEAKREFS(Py_TYPE(ob)). So I can (and did) write a C-extension
> that expose such a function, but it does seem silly that no one else did that
> before me, and that no one seemed to have encountered the problem before.
> 
> So am I missing something ? Is such a function not useful ?  Is there any
> reason not to have it in the stdlib ?

You could try raising a feature request on the bug tracker.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From rosuav at gmail.com  Tue Dec 13 01:05:13 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 13 Dec 2016 17:05:13 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <584f8b08$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <mailman.37.1481585045.2322.python-list@python.org>
 <584f8b08$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoo+m1bZBLPLxf7idDij7PyaWEP0=JW_eN8Rr-9Dao-yA@mail.gmail.com>

On Tue, Dec 13, 2016 at 4:45 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> I don't understand the point of bringing up Javascript. Ben has already said
> that we shouldn't feel the need to mindlessly copy C++ terminology. Is it your
> position that we *should* copy Javascript terminology? Why Javascript and not
> C++ or ObjectiveC? Even when it goes against the obvious English mnemonic?

I'm saying that ALL these terminology debates are needlessly pedantic.
Whatever word you use, you're going to have to explain the Python
semantics as distinct from everyone else's, so don't sweat it. Whether
we call __init__ the constructor or initializer, there is going to be
someone out there who misinterprets it. Go with whatever, pick the
easiest to explain (which is probably "__new__ is allocator, __init__
is initializer"), and don't try to burden the terms alone with the job
of explaining Python's semantics.

ChrisA


From greg.ewing at canterbury.ac.nz  Tue Dec 13 01:14:30 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 13 Dec 2016 19:14:30 +1300
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <5e175c03-4afe-4c46-a3fe-86dd3ef6f583@googlegroups.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <eb8j88Fco3aU1@mid.individual.net>
 <5e175c03-4afe-4c46-a3fe-86dd3ef6f583@googlegroups.com>
Message-ID: <eb9hu6Fjaf6U1@mid.individual.net>

Ned Batchelder wrote:
> if a C++ constructor raises an exception, will the corresponding destructor
> be run, or not? (No, because it never finished making an object of type T.)

So it just leaks any memory that's been allocated by
the partially-run constructor?

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Dec 13 01:21:45 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 13 Dec 2016 19:21:45 +1300
Subject: Method to know if object support being weakreferenced ?
In-Reply-To: <f1f4b779-bb8b-4b57-9413-06861dff121c@googlegroups.com>
References: <f1f4b779-bb8b-4b57-9413-06861dff121c@googlegroups.com>
Message-ID: <eb9ibpFjd6rU1@mid.individual.net>

Matthias Bussonnier wrote:
> I search for a method
> capable of telling me whether an object can be weakreferenced.

Objects that can be weakly referenced have a __weakref__ attribute.
So you could use hasattr(obj, '__weakref__').

-- 
Greg


From vek.m1234 at gmail.com  Tue Dec 13 02:13:31 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Tue, 13 Dec 2016 12:43:31 +0530
Subject: Nested functions, how do they work (stack related)
Message-ID: <o2o70i$c6p$1@dont-email.me>

I was reading the wiki on 'Call stack' because I wanted to understand 
what a traceback object was. My C/C++ isn't good enough to deal with raw 
python source since I have no background in CS. Also, you just can't 
dive into the python src - it takes a good deal of reading and 
background.. (the types will be confusing for a start)

https://en.wikipedia.org/wiki/Call_stack

'Programming languages that support nested subroutines also have a field 
in the call frame that points to the stack frame of the latest 
activation of the procedure that most closely encapsulates the callee, 
i.e. the immediate scope of the callee. This is called an access link or 
static link (as it keeps track of static nesting during dynamic and 
recursive calls) and provides the routine (as well as any other routines 
it may invoke) access to the local data of its encapsulating routines at 
every nesting level. 

Some architectures, compilers, or optimization cases store one link for 
each enclosing level (not just the immediately enclosing), so that 
deeply nested routines that access shallow data do not have to traverse 
several links; this strategy is often called a "display".'

1. What is the difference between a 'call frame' and a 'stack frame' in 
the above context? I know that a stack frame is all the data related to 
one - CALL foo; 

2. He's saying that within the 'call frame' (whatever that is) there's 
an address to one of the previous stack frames of the wrapper function ? 
What does all that mean in terms of nested functions? Access link? How 
are nested function stacks setup..

3. What exactly is a traceback object - we know that an instance object 
is a dictionary and some glue logic that allows you to pretend that 
methods are stored within the instance and call using x.sin(self) etc.
But I was reading: pydoc traceback AND:
http://effbot.org/librarybook/traceback.htm

'Extract the raw traceback from the current stack frame'
A stack frame contains (from wiki) the parameters, local variables, next 
instruction address so.. what's a raw traceback - does the default 
exception handler realize 'okay error' and then walk the stack and 
extract data and prettify it for display and build a magical traceback 
object? Is this documented for dummies what exactly it does?
(i know that's what it's doing but I HAVE NO CLUE so.. are there books 
on this)

How exactly does an exception fit in with tracebacks? How does all this 
fit in with nested functions? 

4. When you call a nested function (decorator), it generally returns a 
wrapper function but I thought he was just returning a reference to a 
function object but obviously since it can see it's environment, how is 
the stack being setup?




From e.bagherzadeh72 at gmail.com  Tue Dec 13 02:47:33 2016
From: e.bagherzadeh72 at gmail.com (Elnaz)
Date: Mon, 12 Dec 2016 23:47:33 -0800 (PST)
Subject: IndexError: list index out of range
Message-ID: <f4337924-d74b-46fa-98f8-aff89caa8415@googlegroups.com>

hi
i am begginer in python. I have written a code and given this error:
IndexError: list index out of range

In my program, I have h=32 bits input. i divide this 32 bits to 4*8 block and every 8-block is n. so n=0:7;(h=int(n/4))
I want to rotate 0 to 7 bits for 2 bits: 0,1,2,3,4,5,6,7--->2,3,4,5,6,7
Iwrite this code:
def rottwo(self, X, n, r):
        assert r >= 1
        temp = [None]*n
        for i in range(n-r) :
            temp[i] = X[i+r]
        for i in range(n-r,n) :
            temp[i] = X[i-n+r]
        return temp
this function work correctly. but I also want to rotate 24 to 31 bits for 5 bits: 24,25,26,27,28,29,30,31-->29,30,31,24,25,26,27,28

when I write this code:
def rotfive(self, X, n, r):
       assert r >= 1
       temp = [None]*n
       for i in range(n-r) :
            temp[i+24] = X[i+3*n+r]
       for i in range(n-r,n) :
            temp[i+24] = X[i+2*n+r]
       return temp 
beacase temp is of size n I cannot access index 3*n+i. index on the list temp should be less than equal to n-1 . 
I son't know how I must correct this!!!!!!!!
Is there any one to help me?
thanks in advanse.


From vek.m1234 at gmail.com  Tue Dec 13 03:47:24 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Tue, 13 Dec 2016 14:17:24 +0530
Subject: Nested functions, how do they work (stack related)
References: <o2o70i$c6p$1@dont-email.me>
Message-ID: <o2ocgf$rl3$1@dont-email.me>

Veek M wrote:

> I was reading the wiki on 'Call stack' because I wanted to understand
> what a traceback object was. My C/C++ isn't good enough to deal with
> raw python source since I have no background in CS. Also, you just
> can't dive into the python src - it takes a good deal of reading and
> background.. (the types will be confusing for a start)
> 
> https://en.wikipedia.org/wiki/Call_stack
> 
> 'Programming languages that support nested subroutines also have a
> field in the call frame that points to the stack frame of the latest
> activation of the procedure that most closely encapsulates the callee,
> i.e. the immediate scope of the callee. This is called an access link
> or static link (as it keeps track of static nesting during dynamic and
> recursive calls) and provides the routine (as well as any other
> routines it may invoke) access to the local data of its encapsulating
> routines at every nesting level.
> 
> Some architectures, compilers, or optimization cases store one link
> for each enclosing level (not just the immediately enclosing), so that
> deeply nested routines that access shallow data do not have to
> traverse several links; this strategy is often called a "display".'
> 
> 1. What is the difference between a 'call frame' and a 'stack frame'
> in the above context? I know that a stack frame is all the data
> related to one - CALL foo;
> 
> 2. He's saying that within the 'call frame' (whatever that is) there's
> an address to one of the previous stack frames of the wrapper function
> ? What does all that mean in terms of nested functions? Access link?
> How are nested function stacks setup..
> 
> 3. What exactly is a traceback object - we know that an instance
> object is a dictionary and some glue logic that allows you to pretend
> that methods are stored within the instance and call using x.sin(self)
> etc. But I was reading: pydoc traceback AND:
> http://effbot.org/librarybook/traceback.htm
> 
> 'Extract the raw traceback from the current stack frame'
> A stack frame contains (from wiki) the parameters, local variables,
> next instruction address so.. what's a raw traceback - does the
> default exception handler realize 'okay error' and then walk the stack
> and extract data and prettify it for display and build a magical
> traceback object? Is this documented for dummies what exactly it does?
> (i know that's what it's doing but I HAVE NO CLUE so.. are there books
> on this)
> 
> How exactly does an exception fit in with tracebacks? How does all
> this fit in with nested functions?
> 
> 4. When you call a nested function (decorator), it generally returns a
> wrapper function but I thought he was just returning a reference to a
> function object but obviously since it can see it's environment, how
> is the stack being setup?

found this:
http://www.drdobbs.com/cpp/how-nested-functions-work-part-1/228701476
(still reading it)


From __peter__ at web.de  Tue Dec 13 04:15:03 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 13 Dec 2016 10:15:03 +0100
Subject: IndexError: list index out of range
References: <f4337924-d74b-46fa-98f8-aff89caa8415@googlegroups.com>
Message-ID: <o2oe6n$6ef$1@blaine.gmane.org>

Elnaz wrote:

> hi
> i am begginer in python. I have written a code and given this error:
> IndexError: list index out of range
> 
> In my program, I have h=32 bits input. i divide this 32 bits to 4*8 block
> and every 8-block is n. so n=0:7;(h=int(n/4)) I want to rotate 0 to 7 bits
> for 2 bits: 0,1,2,3,4,5,6,7--->2,3,4,5,6,7 Iwrite this code:
> def rottwo(self, X, n, r):
>         assert r >= 1
>         temp = [None]*n
>         for i in range(n-r) :
>             temp[i] = X[i+r]
>         for i in range(n-r,n) :
>             temp[i] = X[i-n+r]
>         return temp
> this function work correctly. but I also want to rotate 24 to 31 bits for
> 5 bits: 24,25,26,27,28,29,30,31-->29,30,31,24,25,26,27,28
> 
> when I write this code:
> def rotfive(self, X, n, r):
>        assert r >= 1
>        temp = [None]*n
>        for i in range(n-r) :
>             temp[i+24] = X[i+3*n+r]
>        for i in range(n-r,n) :
>             temp[i+24] = X[i+2*n+r]
>        return temp
> beacase temp is of size n I cannot access index 3*n+i. index on the list
> temp should be less than equal to n-1 . I son't know how I must correct
> this!!!!!!!! Is there any one to help me?
> thanks in advanse.

I think you are making this harder than necessary. Python slices make 
accessing parts of a list quite elegant:

>>> items
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
>>> items[2:5]
[20, 30, 40]
>>> items[3:]
[30, 40, 50, 60, 70, 80, 90]

You can use this to implement a function that creates a rotated list with an 
arbitrary offset:

>>> def rot(items, offset):
...     return items[offset:] + items[:offset]
... 
>>> rot(items, 2)
[20, 30, 40, 50, 60, 70, 80, 90, 0, 10]
>>> rot(items, 7)
[70, 80, 90, 0, 10, 20, 30, 40, 50, 60]
>>> rot(items, -2)
[80, 90, 0, 10, 20, 30, 40, 50, 60, 70]

To rotate part of a list extract that part using slice notation, rotate it 
and write it back:

>>> def rot_part(items, offset, start, stop):
...     items = list(items)
...     items[start:stop] = rot(items[start:stop], offset)
...     return items
... 
>>> rot_part(range(32), 5, 24, 32)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 
21, 22, 23, 29, 30, 31, 24, 25, 26, 27, 28]

If you pass a list as the first argument

items = list(items) 

makes of copy of the list, but it will also convert an arbitrary iterable to 
a list. That's why I can pass the range object.



From marko at pacujo.net  Tue Dec 13 05:46:51 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 13 Dec 2016 12:46:51 +0200
Subject: Nested functions, how do they work (stack related)
References: <o2o70i$c6p$1@dont-email.me>
Message-ID: <87fulsyulg.fsf@elektro.pacujo.net>

Veek M <vek.m1234 at gmail.com>:

> https://en.wikipedia.org/wiki/Call_stack
>
> 'Programming languages that support nested subroutines also have a field 
> in the call frame that points to the stack frame of the latest 
> activation of the procedure that most closely encapsulates the callee, 
> i.e. the immediate scope of the callee. This is called an access link or 
> static link (as it keeps track of static nesting during dynamic and 
> recursive calls) and provides the routine (as well as any other routines 
> it may invoke) access to the local data of its encapsulating routines at 
> every nesting level. 
>
> Some architectures, compilers, or optimization cases store one link for 
> each enclosing level (not just the immediately enclosing), so that 
> deeply nested routines that access shallow data do not have to traverse 
> several links; this strategy is often called a "display".'
>
> 1. What is the difference between a 'call frame' and a 'stack frame' in 
> the above context?

There's no difference.

> 2. He's saying that within the 'call frame' (whatever that is) there's 
> an address to one of the previous stack frames of the wrapper function ? 
> What does all that mean in terms of nested functions? Access link? How 
> are nested function stacks setup..

The classic C stack frame contains two addresses (in addition to the
arguments and local variables):

 * the return address in the calling function

 * the frame pointer in the calling function

Some languages (notably Pascal) add a third address:

 * the frame pointer in the outer function

Often, the outer function is the same as the calling function. However,
if the inner functions call each other, the outer function may be
further up the stack. Since the outer function's local variables are
seen by the inner functions, the extra pointer is needed to access them
directly.

Python has nested functions. Thus, the same technique can be used to
implement Python's internal call stack.

> 3. What exactly is a traceback object

It is an object that reports details of the call stack. It is mostly
useful for troubleshooting.

> How exactly does an exception fit in with tracebacks? How does all this 
> fit in with nested functions?

Well, the traceback object contains also the exception since it is
essential for troubleshooting.

> 4. When you call a nested function (decorator), it generally returns a 
> wrapper function but I thought he was just returning a reference to a 
> function object but obviously since it can see it's environment, how is 
> the stack being setup?

Now I don't exactly understand your question.


Marko


From vek.m1234 at gmail.com  Tue Dec 13 06:32:05 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Tue, 13 Dec 2016 17:02:05 +0530
Subject: Nested functions, how do they work (stack related)
References: <o2o70i$c6p$1@dont-email.me> <o2ocgf$rl3$1@dont-email.me>
Message-ID: <o2om55$vi8$1@dont-email.me>

http://web.archive.org/web/20111030134120/http://www.sidhe.org/~dan/blog/archives/000211.html
(great tail recursion article - best i've seen! SO doesn't really 
explain it unless you already knew it to begin with, but here's the 
link:http://stackoverflow.com/questions/310974/what-is-tail-call-optimization)

I found it useful to read because it deals with the stack. Basically 
when an exception occurs you need to mess with the stack so..


From antoon.pardon at rece.vub.ac.be  Tue Dec 13 06:44:39 2016
From: antoon.pardon at rece.vub.ac.be (Antoon Pardon)
Date: Tue, 13 Dec 2016 12:44:39 +0100
Subject: Nested functions, how do they work (stack related)
In-Reply-To: <o2o70i$c6p$1@dont-email.me>
References: <o2o70i$c6p$1@dont-email.me>
Message-ID: <ba6ffe4b-2034-6b75-e6c0-d4a564ab6f1c@rece.vub.ac.be>

Op 13-12-16 om 08:13 schreef Veek M:
> 4. When you call a nested function (decorator), it generally returns a 
> wrapper function but I thought he was just returning a reference to a 
> function object but obviously since it can see it's environment, how is 
> the stack being setup?

Here you are no longer just talking about nested functions, you are talking
about closures. A stack is no longer sufficient for implementing closures.
The environment for the nested variables of the closure is often alloceted
on the heap.

-- 
Antoon Pardon.



From vek.m1234 at gmail.com  Tue Dec 13 07:11:53 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Tue, 13 Dec 2016 17:41:53 +0530
Subject: Nested functions, how do they work (stack related)
References: <o2o70i$c6p$1@dont-email.me> <87fulsyulg.fsf@elektro.pacujo.net>
Message-ID: <o2oofp$b3o$1@dont-email.me>

Marko Rauhamaa wrote:

> Veek M <vek.m1234 at gmail.com>:
> 
>> https://en.wikipedia.org/wiki/Call_stack
>>
>> 'Programming languages that support nested subroutines also have a
>> field in the call frame that points to the stack frame of the latest
>> activation of the procedure that most closely encapsulates the
>> callee, i.e. the immediate scope of the callee. This is called an
>> access link or static link (as it keeps track of static nesting
>> during dynamic and recursive calls) and provides the routine (as well
>> as any other routines it may invoke) access to the local data of its
>> encapsulating routines at every nesting level.
>>
>> Some architectures, compilers, or optimization cases store one link
>> for each enclosing level (not just the immediately enclosing), so
>> that deeply nested routines that access shallow data do not have to
>> traverse several links; this strategy is often called a "display".'
>>
>> 1. What is the difference between a 'call frame' and a 'stack frame'
>> in the above context?
> 
> There's no difference.
> 
>> 2. He's saying that within the 'call frame' (whatever that is)
>> there's an address to one of the previous stack frames of the wrapper
>> function ? What does all that mean in terms of nested functions?
>> Access link? How are nested function stacks setup..
> 
> The classic C stack frame contains two addresses (in addition to the
> arguments and local variables):
> 
>  * the return address in the calling function
> 
>  * the frame pointer in the calling function
> 
> Some languages (notably Pascal) add a third address:
> 
>  * the frame pointer in the outer function
> 
> Often, the outer function is the same as the calling function.
> However, if the inner functions call each other, the outer function
> may be further up the stack. Since the outer function's local
> variables are seen by the inner functions, the extra pointer is needed
> to access them directly.
> 
> Python has nested functions. Thus, the same technique can be used to
> implement Python's internal call stack.
> 
>> 3. What exactly is a traceback object
> 
> It is an object that reports details of the call stack. It is mostly
> useful for troubleshooting.
> 
>> How exactly does an exception fit in with tracebacks? How does all
>> this fit in with nested functions?
> 
> Well, the traceback object contains also the exception since it is
> essential for troubleshooting.
> 
>> 4. When you call a nested function (decorator), it generally returns
>> a wrapper function but I thought he was just returning a reference to
>> a function object but obviously since it can see it's environment,
>> how is the stack being setup?
> 
> Now I don't exactly understand your question.
> 
> 
> Marko

Umm.. here's an article on windows exception handling.. i was hoping for 
something like that.. (it's very badly written but informative about 
win32 exception handling - i'm still reading it) wanted something 
similar.. I'll quote the highlights.. I KNOW NOTHING about ANY exception 
handling so..

http://www.codeproject.com/KB/cpp/exceptionhandler.aspx

' On the Intel Win32 platform, the FS register always points to the
current TIB. Thus, at FS:[0] you can find a pointer to an 
EXCEPTION_REGISTRATION structure. Now I'm getting somewhere! When an 
exception occurs, the system looks at the TIB of the faulting
thread and retrieves a pointer to an EXCEPTION_REGISTRATION structure. 
In this structure is a pointer to an _except_handler callback function.
The operating system now knows enough to call the _except_handler 
function,'

'When you use a compiler's _try/_except syntax, the compiler also builds 
the EXCEPTION_REGISTRATION struct on the stack. I'm simply showing you a 
simplified version of what a compiler would do if you used 
_try/_except.'






From space.ship.traveller at gmail.com  Tue Dec 13 07:39:26 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Tue, 13 Dec 2016 04:39:26 -0800 (PST)
Subject: Running python from pty without prompt
In-Reply-To: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>

Michael, yes.

FYI, I found out why this works. Pressing Ctrl-D flushes the input
buffer. If you do this on an empty line, it causes read(...) to return
0 which Ruby considers end of input for the script, but the pipe is
not closed.


From frank at chagford.com  Tue Dec 13 08:15:55 2016
From: frank at chagford.com (Frank Millman)
Date: Tue, 13 Dec 2016 15:15:55 +0200
Subject: asyncio question
Message-ID: <o2osa8$uh8$1@blaine.gmane.org>

Hi all

I had a problem with asyncio - not a programming problem, but one with 
organising my code to achieve a given result.

I have come up with a solution, but thought I would mention it here to see 
if there is a better approach.

I am using asyncio.start_server() to run a simple HTTP server. Each request 
is passed to a handler. I strive to complete each request as quickly as 
possible, but I use 'await' where necessary to prevent blocking. It all 
works well.

HTTP does not keep the connection open, it sends a message, waits for a 
response, and closes the connection. In order to maintain 'state' for 
concurrent users, I have a Session class, and each message contains a 
session id so that I can pass the message to the correct session instance. 
Again, it all works well.

The client uses AJAX to send messages to the server. It sends the message 
and continues processing, while a background task waits for the response and 
handles it appropriately. As a result, the client can send a second message 
before receiving a response to the first one. The server can detect this, 
but it cannot wait for the first message to complete, otherwise it will 
block other clients. I have not noticed any problems with processing 2 
requests from the same client concurrently, but I don't like it, so I want 
to process them sequentially.

Here is my solution. As I create each Session instance, I set up a 
background task, using asyncio.ensure_future, which sets up an 
asyncio.Queue. The request handler identifies the session that the request 
belongs to, and 'puts' the request onto that session's Queue. The background 
task runs a 'while True' loop waiting for requests. As they come in it 
'gets' them and processes them. It seems to work.

This means that I have a background task running for each concurrent user. 
Each one will be idle most of the time. My gut-feel says that this will not 
cause a problem, even if there are hundreds of them, but any comments will 
be welcome.

Thanks

Frank Millman




From ian.g.kelly at gmail.com  Tue Dec 13 10:37:34 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 13 Dec 2016 08:37:34 -0700
Subject: asyncio question
In-Reply-To: <o2osa8$uh8$1@blaine.gmane.org>
References: <o2osa8$uh8$1@blaine.gmane.org>
Message-ID: <CALwzid=vdczAH18mHKaL7ryvDUB=7_y-JVUrTkRZ=Gkz66PV3g@mail.gmail.com>

On Tue, Dec 13, 2016 at 6:15 AM, Frank Millman <frank at chagford.com> wrote:
> The client uses AJAX to send messages to the server. It sends the message
> and continues processing, while a background task waits for the response and
> handles it appropriately. As a result, the client can send a second message
> before receiving a response to the first one. The server can detect this,
> but it cannot wait for the first message to complete, otherwise it will
> block other clients. I have not noticed any problems with processing 2
> requests from the same client concurrently, but I don't like it, so I want
> to process them sequentially.

Is there a particular reason why you're worried about this? The
browser is perfectly capable of keeping the requests straight. Also,
note that since the requests use separate HTTP connections, even if
the server sends its responses in a particular order, there's no
guarantee that the client will read them in that order, so this
doesn't free you from the need to allow the client to handle the
requests coming back in any order.

> Here is my solution. As I create each Session instance, I set up a
> background task, using asyncio.ensure_future, which sets up an
> asyncio.Queue. The request handler identifies the session that the request
> belongs to, and 'puts' the request onto that session's Queue. The background
> task runs a 'while True' loop waiting for requests. As they come in it
> 'gets' them and processes them. It seems to work.
>
> This means that I have a background task running for each concurrent user.
> Each one will be idle most of the time. My gut-feel says that this will not
> cause a problem, even if there are hundreds of them, but any comments will
> be welcome.

In a 64-bit Linux build of CPython, the combined size of a generator
and a stack frame is around half a kilobyte (not including whatever
space is needed for local variables), so hundreds of yielding asyncio
tasks should consume no more than hundreds of kilobytes of memory.
Remember that half a kilobyte figure is per generator, not per task,
so if your while loop is four coroutines deep, that will inflate the
cost of the task to two kilobytes each. This is different from the
threading model where each thread would need its own separate stack
space, not just a frame on the heap; you probably wouldn't want to do
this with threads, but with coroutines it should be fine.


From torriem at gmail.com  Tue Dec 13 11:01:22 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 13 Dec 2016 09:01:22 -0700
Subject: Running python from pty without prompt
In-Reply-To: <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
Message-ID: <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>

On 12/13/2016 05:39 AM, Samuel Williams wrote:
> Michael, yes.
> 
> FYI, I found out why this works. Pressing Ctrl-D flushes the input
> buffer. If you do this on an empty line, it causes read(...) to return
> 0 which Ruby considers end of input for the script, but the pipe is
> not closed.

Currently Python does not appear to support this behavior.  Possibly it
could be patched to support something similar, though.



From george.trojan at noaa.gov  Tue Dec 13 11:45:34 2016
From: george.trojan at noaa.gov (George Trojan - NOAA Federal)
Date: Tue, 13 Dec 2016 16:45:34 +0000
Subject: Splitting text into lines
Message-ID: <CABie7_pWf96K8GW1MYrkpxu29xQE0viXxN4iiJ=Jq9gjCvjuzQ@mail.gmail.com>

I have files containing ASCII text with line s separated by '\r\r\n'.
Example:

$ od -c FTAK31_PANC_131140.1481629265635
0000000   F   T   A   K   3   1       P   A   N   C       1   3   1   1
0000020   4   0  \r  \r  \n   T   A   F   A   B   E  \r  \r  \n   T   A
0000040   F  \r  \r  \n   P   A   B   E       1   3   1   1   4   0   Z
0000060       1   3   1   2   /   1   4   1   2       0   7   0   1   0
0000100   K   T       P   6   S   M       S   C   T   0   3   5       O
0000120   V   C   0   6   0  \r  \r  \n                       F   M   1
0000140   3   2   1   0   0       1   0   0   1   2   G   2   0   K   T
0000160       P   6   S   M       B   K   N   1   0   0       W   S   0
0000200   1   5   /   1   8   0   3   5   K   T  \r  \r  \n
0000220           F   M   1   4   1   0   0   0       0   9   0   1   5
0000240   G   2   5   K   T       P   6   S   M       B   K   N   0   5
0000260   0       W   S   0   1   5   /   1   8   0   4   0   K   T   =
0000300  \r  \r  \n
0000303

What is the proper way of getting a list of lines?
Both
>>> open('FTAK31_PANC_131140.1481629265635').readlines()
['FTAK31 PANC 131140\n', '\n', 'TAFABE\n', '\n', 'TAF\n', '\n', 'PABE
131140Z 1312/1412 07010KT P6SM SCT035 OVC060\n', '\n', '     FM132100
10012G20KT P6SM BKN100 WS015/18035KT\n', '\n', '     FM141000 09015G25KT
P6SM BKN050 WS015/18040KT=\n', '\n']

and

>>> open('FTAK31_PANC_131140.1481629265635').read().splitlines()
['FTAK31 PANC 131140', '', 'TAFABE', '', 'TAF', '', 'PABE 131140Z 1312/1412
07010KT P6SM SCT035 OVC060', '', '     FM132100 10012G20KT P6SM BKN100
WS015/18035KT', '', '     FM141000 09015G25KT P6SM BKN050 WS015/18040KT=',
'']

introduce empty (or single character '\n') strings. I can do this:

>>> [x.rstrip() for x in open('FTAK31_PANC_131140.1481629265635',
'rb').read().decode().split('\n')]
['FTAK31 PANC 131140', 'TAFABE', 'TAF', 'PABE 131140Z 1312/1412 07010KT
P6SM SCT035 OVC060', '     FM132100 10012G20KT P6SM BKN100 WS015/18035KT',
'     FM141000 09015G25KT P6SM BKN050 WS015/18040KT=', '']

but it looks cumbersome. I Python2.x I stripped '\r' before passing the
string to split():

>>> open('FTAK31_PANC_131140.1481629265635').read().replace('\r', '')
'FTAK31 PANC 131140\nTAFABE\nTAF\nPABE 131140Z 1312/1412 07010KT P6SM
SCT035 OVC060\n     FM132100 10012G20KT P6SM BKN100 WS015/18035KT\n
FM141000 09015G25KT P6SM BKN050 WS015/18040KT=\n'

but Python 3.x replaces '\r\r\n' by '\n\n' on read().

Ideally I'd like to have code that handles both '\r\r\n' and '\n' as the
split character.

George


From tomuxiong at gmx.com  Tue Dec 13 12:00:33 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Tue, 13 Dec 2016 09:00:33 -0800
Subject: Splitting text into lines
In-Reply-To: <CABie7_pWf96K8GW1MYrkpxu29xQE0viXxN4iiJ=Jq9gjCvjuzQ@mail.gmail.com>
References: <CABie7_pWf96K8GW1MYrkpxu29xQE0viXxN4iiJ=Jq9gjCvjuzQ@mail.gmail.com>
Message-ID: <a9e39a78-d528-83d3-d46b-91b4a281053b@gmx.com>

On 12/13/2016 08:45 AM, George Trojan - NOAA Federal wrote:
> Ideally I'd like to have code that handles both '\r\r\n' and '\n' as the
> split character.
>
> George
>
Are repeated newlines/carriage returns significant at all? What about 
just using re and just replacing any repeated instances of '\r' or '\n' 
with '\n'? I.e. something like

 >>> # the_string is your file all read in
 >>> import re
 >>> re.sub("[\r\n]+", "\n", the_string)

and then continuing as before (i.e. splitting by newlines, etc.)

Does that work?

Cheers,
Thomas


From __peter__ at web.de  Tue Dec 13 12:02:02 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 13 Dec 2016 18:02:02 +0100
Subject: Splitting text into lines
References: <CABie7_pWf96K8GW1MYrkpxu29xQE0viXxN4iiJ=Jq9gjCvjuzQ@mail.gmail.com>
Message-ID: <o2p9i7$bad$1@blaine.gmane.org>

George Trojan - NOAA Federal wrote:

> I have files containing ASCII text with line s separated by '\r\r\n'.

> but it looks cumbersome. I Python2.x I stripped '\r' before passing the
> string to split():
> 
>>>> open('FTAK31_PANC_131140.1481629265635').read().replace('\r', '')
> 'FTAK31 PANC 131140\nTAFABE\nTAF\nPABE 131140Z 1312/1412 07010KT P6SM
> SCT035 OVC060\n     FM132100 10012G20KT P6SM BKN100 WS015/18035KT\n
> FM141000 09015G25KT P6SM BKN050 WS015/18040KT=\n'
> 
> but Python 3.x replaces '\r\r\n' by '\n\n' on read().

Tell Python to keep the newline chars as seen with

open(filename, newline="")

For example:

>>> open("odd-newlines.txt", "rb").read()
b'alpha\nbeta\r\r\ngamma\r\r\ndelta\n'

>>> open("odd-newlines.txt", "r", newline="").read().replace("\r", 
"").splitlines()
['alpha', 'beta', 'gamma', 'delta']

> 
> Ideally I'd like to have code that handles both '\r\r\n' and '\n' as the
> split character.




From torriem at gmail.com  Tue Dec 13 12:12:31 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 13 Dec 2016 10:12:31 -0700
Subject: Running python from pty without prompt
In-Reply-To: <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
Message-ID: <646bbda8-e9a2-886a-d528-0d1a49e452df@gmail.com>

On 12/13/2016 09:01 AM, Michael Torrie wrote:
> On 12/13/2016 05:39 AM, Samuel Williams wrote:
>> Michael, yes.
>>
>> FYI, I found out why this works. Pressing Ctrl-D flushes the input
>> buffer. If you do this on an empty line, it causes read(...) to return
>> 0 which Ruby considers end of input for the script, but the pipe is
>> not closed.
> 
> Currently Python does not appear to support this behavior.  Possibly it
> could be patched to support something similar, though.

I wonder if you could write a python wrapper that would read the input
file from standard in until you get a ctrl-d, then exec() that input.




From george.trojan at noaa.gov  Tue Dec 13 12:25:37 2016
From: george.trojan at noaa.gov (George Trojan - NOAA Federal)
Date: Tue, 13 Dec 2016 17:25:37 +0000
Subject: Splitting text into lines
Message-ID: <CABie7_qv_3_hAsB3xqUznVUn4cDVZve0VuqyEyfgDBf7m0XH0w@mail.gmail.com>

>
> Are repeated newlines/carriage returns significant at all? What about
> just using re and just replacing any repeated instances of '\r' or '\n'
> with '\n'? I.e. something like
>  >>> # the_string is your file all read in
>  >>> import re
>  >>> re.sub("[\r\n]+", "\n", the_string)
> and then continuing as before (i.e. splitting by newlines, etc.)
> Does that work?
> Cheers,
> Thomas


The '\r\r\n' string is a line separator, though not used consistently in US
meteorological bulletins. I do not want to eliminate "real" empty lines.
I was hoping there is a way to prevent read() from making hidden changes to
the file content.

George


From george.trojan at noaa.gov  Tue Dec 13 12:39:24 2016
From: george.trojan at noaa.gov (George Trojan - NOAA Federal)
Date: Tue, 13 Dec 2016 17:39:24 +0000
Subject: Splitting text into lines
Message-ID: <CABie7_o50LcZNve7gZY+1em+n5Wk3skDf=MscJY5qDNRjy3evA@mail.gmail.com>

>
> Tell Python to keep the newline chars as seen with
> open(filename, newline="")
> For example:
> >>>
> * open("odd-newlines.txt", "rb").read() *
> b'alpha\nbeta\r\r\ngamma\r\r\ndelta\n'
> >>>
> * open("odd-newlines.txt", "r", newline="").read().replace("\r", *
> "").splitlines()
> ['alpha', 'beta', 'gamma', 'delta']


Thanks Peter. That's what I needed.

George


From random832 at fastmail.com  Tue Dec 13 12:48:07 2016
From: random832 at fastmail.com (Random832)
Date: Tue, 13 Dec 2016 12:48:07 -0500
Subject: Running python from pty without prompt
In-Reply-To: <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
Message-ID: <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>

On Tue, Dec 13, 2016, at 11:01, Michael Torrie wrote:
> On 12/13/2016 05:39 AM, Samuel Williams wrote:
> > Michael, yes.
> > 
> > FYI, I found out why this works. Pressing Ctrl-D flushes the input
> > buffer. If you do this on an empty line, it causes read(...) to return
> > 0 which Ruby considers end of input for the script, but the pipe is
> > not closed.
>
> Currently Python does not appear to support this behavior.  Possibly it
> could be patched to support something similar, though.

The problem is there's currently no way to differentiate "interactive
mode" from "script run on a tty".

You can get similar behavior with python -c "import
sys;exec(sys.stdin.read())"


From random832 at fastmail.com  Tue Dec 13 12:54:51 2016
From: random832 at fastmail.com (Random832)
Date: Tue, 13 Dec 2016 12:54:51 -0500
Subject: Splitting text into lines
In-Reply-To: <CABie7_qv_3_hAsB3xqUznVUn4cDVZve0VuqyEyfgDBf7m0XH0w@mail.gmail.com>
References: <CABie7_qv_3_hAsB3xqUznVUn4cDVZve0VuqyEyfgDBf7m0XH0w@mail.gmail.com>
Message-ID: <1481651691.139394.817801841.73CB0C40@webmail.messagingengine.com>

On Tue, Dec 13, 2016, at 12:25, George Trojan - NOAA Federal wrote:
> >
> > Are repeated newlines/carriage returns significant at all? What about
> > just using re and just replacing any repeated instances of '\r' or '\n'
> > with '\n'? I.e. something like
> >  >>> # the_string is your file all read in
> >  >>> import re
> >  >>> re.sub("[\r\n]+", "\n", the_string)
> > and then continuing as before (i.e. splitting by newlines, etc.)
> > Does that work?
> > Cheers,
> > Thomas
> 
> 
> The '\r\r\n' string is a line separator, though not used consistently in
> US
> meteorological bulletins. I do not want to eliminate "real" empty lines.

I'd do re.sub("\r*\n", "\n", the_string). Any "real" empty lines are
almost certainly going to have two \n characters, regardless of any \r
characters. It looks like what *happens* is that the file, or some part
of the file, had \r\n line endings originally and was "converted" to
turn the \n into \r\n.

> I was hoping there is a way to prevent read() from making hidden changes
> to the file content.

Pass newline='' into open.


From paolieri at gmail.com  Tue Dec 13 15:27:26 2016
From: paolieri at gmail.com (paolieri at gmail.com)
Date: Tue, 13 Dec 2016 12:27:26 -0800 (PST)
Subject: Name mangling vs qualified access to class attributes
Message-ID: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>

The official Python tutorial at

https://docs.python.org/3/tutorial/classes.html#private-variables

says that "name mangling is helpful for letting subclasses override methods without breaking intraclass method calls" and makes an interesting example:

class Mapping:
    def __init__(self, iterable):
        self.items_list = []
        self.__update(iterable)

    def update(self, iterable):
        for item in iterable:
            self.items_list.append(item)

    __update = update   # private copy of original update() method

class MappingSubclass(Mapping):

    def update(self, keys, values):
        # provides new signature for update()
        # but does not break __init__()
        for item in zip(keys, values):
            self.items_list.append(item)


It seems to me that, in this example, one could just have:

class Mapping:
    def __init__(self, iterable):
        self.items_list = []
        Mapping.update(self, iterable)

    def update(self, iterable):
        for item in iterable:
            self.items_list.append(item)

and avoid copying 'Mapping.update' into 'Mapping.__update'. More generally, any time one needs to "let subclasses override methods without breaking intraclass method calls" (the goal stated in the tutorial), using qualified access to class attributes/methods should suffice.

Am I missing something? Is 'self.__update(iterable)' in 'Mapping.__init__' preferable to 'Mapping.update(self, iterable)'?

I think that, instead, name mangling is helpful to avoid accidental overrides of methods/attributes by the *current* class (rather than its subclasses). Given the way that C3 linearization works, you can't know in advance who will follow your class A in B.__mro__ when B extends A. Name mangling allows you to avoid overriding methods/attributes of classes that might follow.

Any thoughts?

Best,
-- Marco


From rbrowning1 at mmm.com  Tue Dec 13 15:37:59 2016
From: rbrowning1 at mmm.com (Rhesa Browning)
Date: Tue, 13 Dec 2016 20:37:59 +0000
Subject: Install Problem
Message-ID: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>

I have been trying to install Python 3.5.2 onto my computer.  I have installed and uninstalled and resinstalled several times.  Every time I get an error message saying that the python35.dll doesn't exist on my computer so it can't open Python.  How can this problem be fixed?



rbrowning1 at mmm.com<mailto:rbrowning1 at mmm.com>



From gordon at panix.com  Tue Dec 13 15:51:49 2016
From: gordon at panix.com (John Gordon)
Date: Tue, 13 Dec 2016 20:51:49 +0000 (UTC)
Subject: Install Problem
References: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
 <mailman.55.1481661714.2322.python-list@python.org>
Message-ID: <o2pn15$dd6$1@reader1.panix.com>

In <mailman.55.1481661714.2322.python-list at python.org> Rhesa Browning <rbrowning1 at mmm.com> writes:

>I have been trying to install Python 3.5.2 onto my computer.  I have
>installed and uninstalled and resinstalled several times.  Every time I
>get an error message saying that the python35.dll doesn't exist on my
>computer so it can't open Python.  How can this problem be fixed?

When does the error occur?  During installation?  Or after installation
when you're trying to run a python program?

What version of Windows do you have?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From rbrowning1 at mmm.com  Tue Dec 13 16:05:18 2016
From: rbrowning1 at mmm.com (Rhesa Browning)
Date: Tue, 13 Dec 2016 21:05:18 +0000
Subject: Install Problem
In-Reply-To: <o2pn15$dd6$1@reader1.panix.com>
References: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
 <mailman.55.1481661714.2322.python-list@python.org>
 <o2pn15$dd6$1@reader1.panix.com>
Message-ID: <DM5PR03MB28900F8632FEA04B93A6AEBF9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>

It occurs when I am trying to open up a python program.  I am running windows 7.

-----Original Message-----
From: Python-list [mailto:python-list-bounces+rbrowning1=mmm.com at python.org] On Behalf Of John Gordon
Sent: Tuesday, December 13, 2016 2:52 PM
To: python-list at python.org
Subject: [EXTERNAL] Re: Install Problem

In <mailman.55.1481661714.2322.python-list at python.org> Rhesa Browning <rbrowning1 at mmm.com> writes:

>I have been trying to install Python 3.5.2 onto my computer.  I have 
>installed and uninstalled and resinstalled several times.  Every time I 
>get an error message saying that the python35.dll doesn't exist on my 
>computer so it can't open Python.  How can this problem be fixed?

When does the error occur?  During installation?  Or after installation when you're trying to run a python program?

What version of Windows do you have?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

--
https://mail.python.org/mailman/listinfo/python-list
3M security scanners have not detected any malicious content in this message.

To report this email as SPAM, please forward it to spam at websense.com


From eryksun at gmail.com  Tue Dec 13 16:48:51 2016
From: eryksun at gmail.com (eryk sun)
Date: Tue, 13 Dec 2016 21:48:51 +0000
Subject: Install Problem
In-Reply-To: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
References: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
Message-ID: <CACL+1as-u5mcBdh2Bd_HPitMM84pKnVZAemc+tgGPe14JNHsVg@mail.gmail.com>

On Tue, Dec 13, 2016 at 8:37 PM, Rhesa Browning <rbrowning1 at mmm.com> wrote:
> I have been trying to install Python 3.5.2 onto my computer.  I have installed and
> uninstalled and resinstalled several times.  Every time I get an error message
> saying that the python35.dll doesn't exist on my computer so it can't open Python.
> How can this problem be fixed?

Please open an issue at bugs.python.org. Zip the installation logs
("Python 3.5.2*.log") from your %temp% folder, and upload them to the
issue.


From cs at zip.com.au  Tue Dec 13 16:52:23 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 14 Dec 2016 08:52:23 +1100
Subject: Name mangling vs qualified access to class attributes
In-Reply-To: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>
References: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>
Message-ID: <20161213215223.GA91248@cskk.homeip.net>

On 13Dec2016 12:27, paolieri at gmail.com <paolieri at gmail.com> wrote:
>The official Python tutorial at
>https://docs.python.org/3/tutorial/classes.html#private-variables
>
>says that "name mangling is helpful for letting subclasses override methods without breaking intraclass method calls" and makes an interesting example:
>
>class Mapping:
>    def __init__(self, iterable):
>        self.items_list = []
>        self.__update(iterable)
>
>    def update(self, iterable):
>        for item in iterable:
>            self.items_list.append(item)
>
>    __update = update   # private copy of original update() method
>
>class MappingSubclass(Mapping):
>
>    def update(self, keys, values):
>        # provides new signature for update()
>        # but does not break __init__()
>        for item in zip(keys, values):
>            self.items_list.append(item)
>
>It seems to me that, in this example, one could just have:
>
>class Mapping:
>    def __init__(self, iterable):
>        self.items_list = []
>        Mapping.update(self, iterable)
>
>    def update(self, iterable):
>        for item in iterable:
>            self.items_list.append(item)
>
>and avoid copying 'Mapping.update' into 'Mapping.__update'. More generally, any time one needs to "let subclasses override methods without breaking intraclass method calls" (the goal stated in the tutorial), using qualified access to class attributes/methods should suffice.
>
>Am I missing something? Is 'self.__update(iterable)' in 'Mapping.__init__' preferable to 'Mapping.update(self, iterable)'?

IMO, mostly in that "Mapping.update" hardwires the class name, whereas 
"self.__update" will survive a class rename.

I confess I've never used name mangling in the manner shown in the example.

Hoping for more insightful comments...

Cheers,
Cameron Simpson <cs at zip.com.au>


From torriem at gmail.com  Tue Dec 13 17:09:19 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 13 Dec 2016 15:09:19 -0700
Subject: Running python from pty without prompt
In-Reply-To: <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
Message-ID: <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>

On 12/13/2016 10:48 AM, Random832 wrote:
> The problem is there's currently no way to differentiate "interactive
> mode" from "script run on a tty".
> 
> You can get similar behavior with python -c "import
> sys;exec(sys.stdin.read())"

Are you sure? I can pipe scripts into Python and they run fine and
Python is not in interactive mode.

python < script.py

The behavior the OP is looking for of course is a way of demarcating the
end of the script and the beginning of data to feed the script.



From random832 at fastmail.com  Tue Dec 13 17:24:57 2016
From: random832 at fastmail.com (Random832)
Date: Tue, 13 Dec 2016 17:24:57 -0500
Subject: Running python from pty without prompt
In-Reply-To: <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
Message-ID: <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>

On Tue, Dec 13, 2016, at 17:09, Michael Torrie wrote:
> On 12/13/2016 10:48 AM, Random832 wrote:
> > The problem is there's currently no way to differentiate "interactive
> > mode" from "script run on a tty".
> > 
> > You can get similar behavior with python -c "import
> > sys;exec(sys.stdin.read())"
> 
> Are you sure? I can pipe scripts into Python and they run fine and
> Python is not in interactive mode.

Yes, a pipe and a tty are two different things.

> python < script.py
> 
> The behavior the OP is looking for of course is a way of demarcating the
> end of the script and the beginning of data to feed the script.

It's more than just that - with a tty you can call sys.stdin.read()
multiple times, and each time end it with ctrl-d.


From steve+python at pearwood.info  Tue Dec 13 19:05:46 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 14 Dec 2016 11:05:46 +1100
Subject: Splitting text into lines
References: <CABie7_pWf96K8GW1MYrkpxu29xQE0viXxN4iiJ=Jq9gjCvjuzQ@mail.gmail.com>
 <mailman.46.1481647544.2322.python-list@python.org>
Message-ID: <58508cdc$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Wed, 14 Dec 2016 03:45 am, George Trojan - NOAA Federal wrote:

> I have files containing ASCII text with line s separated by '\r\r\n'.
> Example:
> 
> $ od -c FTAK31_PANC_131140.1481629265635
> 0000000   F   T   A   K   3   1       P   A   N   C       1   3   1   1
> 0000020   4   0  \r  \r  \n   T   A   F   A   B   E  \r  \r  \n   T   A
[...] 
> 0000300  \r  \r  \n
> 0000303
> 
> What is the proper way of getting a list of lines?

Do you have any objection to post-processing the list of lines?

lines = open(filename).readlines()
# remove leading and trailing whitespace, including newline characters
lines = [line.strip() for line in lines]
# get rid of empty strings
lines = [line for line in lines if line]

If you prefer, that last line of code can be written as either:

lines = filter(None, lines)  # Python 2
lines = list(filter(None, lines))  # Python 3


Personally, this would be my preferred technique, as I (nearly) always end
up doing a strip() on data I read from text files, so the extra call to
filter is no big deal.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Dec 13 19:10:40 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 14 Dec 2016 11:10:40 +1100
Subject: Running python from pty without prompt
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
 <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>
 <mailman.60.1481667900.2322.python-list@python.org>
Message-ID: <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Wed, 14 Dec 2016 09:24 am, Random832 wrote:

> On Tue, Dec 13, 2016, at 17:09, Michael Torrie wrote:
>> On 12/13/2016 10:48 AM, Random832 wrote:
>> > The problem is there's currently no way to differentiate "interactive
>> > mode" from "script run on a tty".
>> > 
>> > You can get similar behavior with python -c "import
>> > sys;exec(sys.stdin.read())"
>> 
>> Are you sure? I can pipe scripts into Python and they run fine and
>> Python is not in interactive mode.
> 
> Yes, a pipe and a tty are two different things.


Can you show a simple demonstration of what you are doing?

I'm having difficulty following this thread because I don't know
what "script run on a tty" means.

I thought that with the exception of scripts run from cron, any time you run
a script *or* in interactive mode, there is an associated tty. Am I wrong?



>> python < script.py
>> 
>> The behavior the OP is looking for of course is a way of demarcating the
>> end of the script and the beginning of data to feed the script.
> 
> It's more than just that - with a tty you can call sys.stdin.read()
> multiple times, and each time end it with ctrl-d.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Dec 13 19:20:35 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 14 Dec 2016 11:20:35 +1100
Subject: Running python from pty without prompt
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <mailman.53.1481651778.2322.python-list@python.org>
Message-ID: <58509055$0$1585$c3e8da3$5496439d@news.astraweb.com>

On Wed, 14 Dec 2016 04:48 am, Random832 wrote:

> On Tue, Dec 13, 2016, at 11:01, Michael Torrie wrote:
>> On 12/13/2016 05:39 AM, Samuel Williams wrote:
>> > Michael, yes.
>> > 
>> > FYI, I found out why this works. Pressing Ctrl-D flushes the input
>> > buffer. If you do this on an empty line, it causes read(...) to return
>> > 0 which Ruby considers end of input for the script, but the pipe is
>> > not closed.
>>
>> Currently Python does not appear to support this behavior.  Possibly it
>> could be patched to support something similar, though.
> 
> The problem is there's currently no way to differentiate "interactive
> mode" from "script run on a tty".


sys.flags.interactive will tell you whether or not your script was launched
with the -i flag.

hasattr(sys, 'ps1') or hasattr(sys, 'ps2') will tell you if you are running
in the REPL (interactive interpreter). The ps1 and ps2 variables aren't
defined in non-interactive mode.

Does that help?


> You can get similar behavior with python -c "import
> sys;exec(sys.stdin.read())"

[steve at ando ~]$ python -c "import sys; print hasattr(sys, 'ps1')"
False

[steve at ando ~]$ python -c "import sys; exec(sys.stdin.read())"
import sys
print hasattr(sys, 'ps1')
False


It's not obvious, but after I entered the line "print hasattr(...)" I typed
Ctrl-D, ending the stream.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From torriem at gmail.com  Tue Dec 13 19:49:11 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 13 Dec 2016 17:49:11 -0700
Subject: Running python from pty without prompt
In-Reply-To: <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
 <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>
 <mailman.60.1481667900.2322.python-list@python.org>
 <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <6f30a1a6-29ea-adeb-525f-f68ae211bbac@gmail.com>

On 12/13/2016 05:10 PM, Steve D'Aprano wrote:
> Can you show a simple demonstration of what you are doing?

I think they want to run Python, perhaps remotely via ssh, and feed it
both a script and input over standard-in (though a tty comes into this
somehow and I'm not clear on that).  Apparently in Ruby you can pass a
script to it via standard-in, then a ctrl-d, and standard-in is kept
open so they can then feed the ruby script input.  If Python supported
this, an example would look something like this:

$ python << EOF
a = input("Give me something: ")
print (a)
<^D>
test_input
EOF

Where ^D is a literal control-d character the marks the end of the
script and the beginning of input that will go to the script.

The tty part might come into play when they are using ssh to remotely
run the python process.  Standard in, though, is the primary mechanism
they want to use if I understand the OP correctly.

I think a wrapper that feeds exec() would do what he desires.



From ben+python at benfinney.id.au  Tue Dec 13 19:54:42 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 14 Dec 2016 11:54:42 +1100
Subject: Running python from pty without prompt
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
 <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>
 <mailman.60.1481667900.2322.python-list@python.org>
 <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <85pokvz5wt.fsf@benfinney.id.au>

Steve D'Aprano <steve+python at pearwood.info> writes:

> I thought that with the exception of scripts run from cron, any time
> you run a script *or* in interactive mode, there is an associated tty.
> Am I wrong?

Any daemon will, by definition, have no controlling terminal.

Other processes can choose to detach themselves from their controlling
terminal.

Either of those could invoke Python, and then the Python program would
be running without any controlling terminal.

-- 
 \       ?I have always wished for my computer to be as easy to use as |
  `\       my telephone; my wish has come true because I can no longer |
_o__)          figure out how to use my telephone.? ?Bjarne Stroustrup |
Ben Finney



From skip.montanaro at gmail.com  Tue Dec 13 20:06:45 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Tue, 13 Dec 2016 19:06:45 -0600
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwc8brRcGE2fotW5-541YfzxgMe3H2FzwvbUPA3Nz40-w@mail.gmail.com>
 <CANc-5Uy+D7dqfjMGZhovQ1aoH8RT1ENvDqEb4SXcrgNoHRwUEA@mail.gmail.com>
 <CANc-5UwXQDQ4c4TekZO9aYEL_fudm7iGziws52AkQHX4s14qHQ@mail.gmail.com>
 <CANc-5Ux6m9hJDeYEZ4V3ExHR=XtTmywmVvaPYY3s9w3s3UqqHA@mail.gmail.com>
 <CANc-5UyAhmA4Oq1C3Rk+aebCB35-h17SbhAC2_YwmH1NNvzF_Q@mail.gmail.com>
 <CANc-5Ux+x6cQnM+7b71RgYN8P1o3RarF53o_0K6TG1sf+5oBjQ@mail.gmail.com>
 <CANc-5UzTO_zaUQeTW0sBtc4_rU1r=JeVQ0SiH-0zsrxnotjqtQ@mail.gmail.com>
 <CANc-5UyPhS+gD-BuxWvQNDw7Wk-VsAa8fK7xMU9Osao9BVdZVQ@mail.gmail.com>
 <CANc-5UxgY3vL1VprybVXCxF_5YgL0cjVDFY_br1TS5nuvxEOkA@mail.gmail.com>
 <CANc-5Uwi+Vp+inGKooN7GSbk=hppQYWOjSNCrbb6EVNgmh+UgA@mail.gmail.com>
 <CANc-5Uy7-aA32y1gLcp=zLnHc-itTb4xOAikqHxgO4mDwx6DcA@mail.gmail.com>
 <CANc-5UyOQopd75p9K6fWmOmtQcdW0Td-O-6R=_ZFVaBQvcKqTg@mail.gmail.com>
 <CANc-5UzaLXqUWUro6nJ-x8VmCtaeCWrwEayGE0X4i9ZtAm0QSw@mail.gmail.com>
 <CANc-5UzMN19xyY07q3DgKXyR9M=C4CYAUhQoXrhL6RuUGMPS=A@mail.gmail.com>
 <CANc-5Uw2ta34basMuG88S4bGG=sGSVwxK4gzm=qAy0DHK-CGDQ@mail.gmail.com>
 <CANc-5UxvefxVYoZqNVVBbEYWbpjnbGk-a964_6Hae2WFNwj7_w@mail.gmail.com>
 <CANc-5Uxw=AScfO0+LVUgL9X29JgH9cqVrJj8=dKd6Q0tc7gLsg@mail.gmail.com>
 <CANc-5Uy1usatpzb3Aq0616TEuQhx93QNgF_xkhboSaPX4E5nVQ@mail.gmail.com>
 <CANc-5UwgHQvqeM5GspNrAEfGu6U2hgry3mmA-ys8Oz9Rs47oiw@mail.gmail.com>
 <CANc-5Uz6o5Cj0U=Nd+Zjihd-NLevV1guo4nUdqx6-5TM4O0QoQ@mail.gmail.com>
 <CANc-5UzE_vmgN+BOz8sFHJuxxpzXmhVf3nVocLvLg1GnRV1zMg@mail.gmail.com>
 <CANc-5UxixW1mKQXmm56d_eGgKcpz0tLkNqdTnC36p3uLUoN+TQ@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
Message-ID: <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>

I know this isn't a Python-specific question, but i got zero useful
responses from the help-gnu-emacs list for some reason. I think this
expression should not evaluate to the empty set:

set(python_programmers) & set(emacs_users) & set(new_macbookpro_owners)

Hopefully there are a few people out there who've had an opportunity to try
the new ESC-less Pros with the Touch Bar for an extended period of time.
Does the lack of a physical ESC key create problems for people, especially
Emacs users?

Just to head off a couple suggestions people might be inclined to make...
Yes, I know I can use C-[ or the Alt key instead of ESC. I can remap other
keys like Caps Lock or the back tick. I can also buy some other laptop with
a true ESC key, or buy a 13-inch MBP sans Touch Bar. I do plan to try out
Emacs on a Touch-Bar-equipped MacBook Pro at the Apple Store, but a few
minutes horsing around in the din of holiday shopping isn't the same as an
extended test drive in my usual environment.

So, for those of you who've tried it, does the lack of a physical ESC key
create problems?

Thx,

Skip


From torriem at gmail.com  Tue Dec 13 21:33:07 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 13 Dec 2016 19:33:07 -0700
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxixW1mKQXmm56d_eGgKcpz0tLkNqdTnC36p3uLUoN+TQ@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
Message-ID: <931bb40b-d58d-39ce-6bc2-ce19e641ba84@gmail.com>

On 12/13/2016 06:06 PM, Skip Montanaro wrote:
> So, for those of you who've tried it, does the lack of a physical ESC key
> create problems?

If there were problems with it I imagine ViM users would probably be
more inconvenienced than Emacs users.  I haven't heard anything about
people's real-world experience. I know of no one with one of these
expensive little machines yet.  The stores around here don't even have
them yet.  I suspect you haven't gotten much feedback because very few
users (Emacs users anyway) have gotten a hold of these new machines yet.

If I was to choose between a 2015 MBP and the new touch bar MBP, no
question I'd just get the 2015 while I still can, and that has nothing
to do with the touch bar.  Way better bang for the computing buck. The
new machines are overpriced now.

But I am getting cynical in my old age.  My love affair with Apple that
started with OS X 10.0 has steadily waned over the years.  Lately I've
actually been impressed with the nicer Windows 10 laptops and tablets.
I'm quite shocked actually (at that fact).




From no.email at nospam.invalid  Tue Dec 13 21:40:21 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Tue, 13 Dec 2016 18:40:21 -0800
Subject: OT - "Soft" ESC key on the new MacBook Pro
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
Message-ID: <8760mn2pyi.fsf@nightsong.com>

Skip Montanaro <skip.montanaro at gmail.com> writes:
> Does the lack of a physical ESC key create problems for people, especially
> Emacs users?

Not a Mac user and I rarely use ESC instead of ALT while editing with
Emacs on a local computer, but when editing remotely I do have to use
ESC because the Gnome terminal emulator steals a few ALTed keys.  Maybe
there is a way to stop that behaviour but it didn't occur to me til just
now.  Hmm.

Meanwhile the concept of a computer with "no escape" just shows Apple
getting deeper into existentialism.  First it was the hipster Mac users
with the Beatnik black berets and turtlenecks, and now this. <wink>


From greg.ewing at canterbury.ac.nz  Tue Dec 13 23:22:32 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Wed, 14 Dec 2016 17:22:32 +1300
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <8760mn2pyi.fsf@nightsong.com>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>	<CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>	<CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>	<CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>	<CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>	<CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>	<CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>	<CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>	<CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>	<CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>	<CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>	<CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>	<CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>	<mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com>
Message-ID: <ebbvoaF750lU1@mid.individual.net>

Paul Rubin wrote:
> First it was the hipster Mac users
> with the Beatnik black berets and turtlenecks, and now this. <wink>

Once you're in the clutches of Apple, there is no Escape.

-- 
Greg


From vek.m1234 at gmail.com  Wed Dec 14 01:11:31 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Wed, 14 Dec 2016 11:41:31 +0530
Subject: Is there a way to insert hooks into a native dictionary type to see
 when a query arrives and what's looked up?
Message-ID: <o2qno3$oqp$1@dont-email.me>

I know that with user classes one can define getattr, setattr to handle 
dictionary lookup. Is there a way to hook into the native dict() type 
and see in real time what's being queried.

I wanted to check if when one does:

x.sin()

if the x.__dict__ was queried or if the Foo.__dict__ was queried.. I 
know method/attribute lookup starts with the instance but was wondering 
if I could see it in action vs defining __getattr__ __setattr__ in Foo 
which is a bit indirect.. and not what I want.


From frank at chagford.com  Wed Dec 14 01:23:39 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 14 Dec 2016 08:23:39 +0200
Subject: asyncio question
In-Reply-To: <CALwzid=vdczAH18mHKaL7ryvDUB=7_y-JVUrTkRZ=Gkz66PV3g@mail.gmail.com>
References: <o2osa8$uh8$1@blaine.gmane.org>
 <CALwzid=vdczAH18mHKaL7ryvDUB=7_y-JVUrTkRZ=Gkz66PV3g@mail.gmail.com>
Message-ID: <o2qoh8$ffi$1@blaine.gmane.org>

"Ian Kelly"  wrote in message 
news:CALwzid=vdczAH18mHKaL7ryvDUB=7_y-JVUrTkRZ=Gkz66PV3g at mail.gmail.com...
>
> On Tue, Dec 13, 2016 at 6:15 AM, Frank Millman <frank at chagford.com> wrote:
> > The client uses AJAX to send messages to the server. It sends the 
> > message
> > and continues processing, while a background task waits for the response 
> > and
> > handles it appropriately. As a result, the client can send a second 
> > message
> > before receiving a response to the first one. The server can detect 
> > this,
> > but it cannot wait for the first message to complete, otherwise it will
> > block other clients. I have not noticed any problems with processing 2
> > requests from the same client concurrently, but I don't like it, so I 
> > want
> > to process them sequentially.
>
> Is there a particular reason why you're worried about this? The
> browser is perfectly capable of keeping the requests straight. Also,
> note that since the requests use separate HTTP connections, even if
> the server sends its responses in a particular order, there's no
> guarantee that the client will read them in that order, so this
> doesn't free you from the need to allow the client to handle the
> requests coming back in any order.
>

I had not thought of that, thanks. In fact, more to the point in my case, I 
assume that there is no guarantee that the server will receive the requests 
in the same order that the client sends them.

The particular reason for my concern was that each request can change the 
state of the session, and I wanted to be sure that the state has been fully 
updated before processing the next request.

One scenario, that I had not thought of, is that the requests could be 
received out of sequence. I don't think this will be a problem, but I will 
have to think about it.

The second scenario, which was my main concern, is that the server starts 
processing the second request before processing of the first request has 
been completed, meaning that the session data may not be in a stable state. 
My proposed solution solves this problem.

>
> In a 64-bit Linux build of CPython, the combined size of a generator
> and a stack frame is around half a kilobyte (not including whatever
> space is needed for local variables), so hundreds of yielding asyncio
> tasks should consume no more than hundreds of kilobytes of memory.
> Remember that half a kilobyte figure is per generator, not per task,
> so if your while loop is four coroutines deep, that will inflate the
> cost of the task to two kilobytes each. This is different from the
> threading model where each thread would need its own separate stack
> space, not just a frame on the heap; you probably wouldn't want to do
> this with threads, but with coroutines it should be fine.
>

Good to know, thanks. I will proceed on the assumption that if anyone runs 
my system with hundreds of users, they will run it on some serious hardware, 
so I should be safe.

Frank




From rosuav at gmail.com  Wed Dec 14 01:29:47 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 14 Dec 2016 17:29:47 +1100
Subject: Is there a way to insert hooks into a native dictionary type to
 see when a query arrives and what's looked up?
In-Reply-To: <o2qno3$oqp$1@dont-email.me>
References: <o2qno3$oqp$1@dont-email.me>
Message-ID: <CAPTjJmpqGst6Da3gvmtB08mH=hdRSHF4Yr4mvU=N62nWQxM5vw@mail.gmail.com>

On Wed, Dec 14, 2016 at 5:11 PM, Veek M <vek.m1234 at gmail.com> wrote:
> I know that with user classes one can define getattr, setattr to handle
> dictionary lookup. Is there a way to hook into the native dict() type
> and see in real time what's being queried.
>
> I wanted to check if when one does:
>
> x.sin()
>
> if the x.__dict__ was queried or if the Foo.__dict__ was queried.. I
> know method/attribute lookup starts with the instance but was wondering
> if I could see it in action vs defining __getattr__ __setattr__ in Foo
> which is a bit indirect.. and not what I want.

Normally, I would say "subclass dict and override __getitem__", but
interestingly, that doesn't seem to work. You can put a subclass of
dict in an object's __dict__, but Python won't call your functions -
at least, CPython 3.7 won't. Presumably the actual lookups and updates
are done by directly tinkering with the dict's internals, from C.

ChrisA


From e.bagherzadeh72 at gmail.com  Wed Dec 14 01:38:06 2016
From: e.bagherzadeh72 at gmail.com (Elnaz)
Date: Tue, 13 Dec 2016 22:38:06 -0800 (PST)
Subject: IndexError: list index out of range
In-Reply-To: <mailman.41.1481620524.2322.python-list@python.org>
References: <f4337924-d74b-46fa-98f8-aff89caa8415@googlegroups.com>
 <o2oe6n$6ef$1@blaine.gmane.org>
 <mailman.41.1481620524.2322.python-list@python.org>
Message-ID: <008ec54a-9ef0-46ba-8cb1-49a52ba40f09@googlegroups.com>

On Tuesday, December 13, 2016 at 12:45:49 PM UTC+3:30, Peter Otten wrote:
> Elnaz wrote:
> 
> > hi
> > i am begginer in python. I have written a code and given this error:
> > IndexError: list index out of range
> > 
> > In my program, I have h=32 bits input. i divide this 32 bits to 4*8 block
> > and every 8-block is n. so n=0:7;(h=int(n/4)) I want to rotate 0 to 7 bits
> > for 2 bits: 0,1,2,3,4,5,6,7--->2,3,4,5,6,7 Iwrite this code:
> > def rottwo(self, X, n, r):
> >         assert r >= 1
> >         temp = [None]*n
> >         for i in range(n-r) :
> >             temp[i] = X[i+r]
> >         for i in range(n-r,n) :
> >             temp[i] = X[i-n+r]
> >         return temp
> > this function work correctly. but I also want to rotate 24 to 31 bits for
> > 5 bits: 24,25,26,27,28,29,30,31-->29,30,31,24,25,26,27,28
> > 
> > when I write this code:
> > def rotfive(self, X, n, r):
> >        assert r >= 1
> >        temp = [None]*n
> >        for i in range(n-r) :
> >             temp[i+24] = X[i+3*n+r]
> >        for i in range(n-r,n) :
> >             temp[i+24] = X[i+2*n+r]
> >        return temp
> > beacase temp is of size n I cannot access index 3*n+i. index on the list
> > temp should be less than equal to n-1 . I son't know how I must correct
> > this!!!!!!!! Is there any one to help me?
> > thanks in advanse.
> 
> I think you are making this harder than necessary. Python slices make 
> accessing parts of a list quite elegant:
> 
> >>> items
> [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
> >>> items[2:5]
> [20, 30, 40]
> >>> items[3:]
> [30, 40, 50, 60, 70, 80, 90]
> 
> You can use this to implement a function that creates a rotated list with an 
> arbitrary offset:
> 
> >>> def rot(items, offset):
> ...     return items[offset:] + items[:offset]
> ... 
> >>> rot(items, 2)
> [20, 30, 40, 50, 60, 70, 80, 90, 0, 10]
> >>> rot(items, 7)
> [70, 80, 90, 0, 10, 20, 30, 40, 50, 60]
> >>> rot(items, -2)
> [80, 90, 0, 10, 20, 30, 40, 50, 60, 70]
> 
> To rotate part of a list extract that part using slice notation, rotate it 
> and write it back:
> 
> >>> def rot_part(items, offset, start, stop):
> ...     items = list(items)
> ...     items[start:stop] = rot(items[start:stop], offset)
> ...     return items
> ... 
> >>> rot_part(range(32), 5, 24, 32)
> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 
> 21, 22, 23, 29, 30, 31, 24, 25, 26, 27, 28]
> 
> If you pass a list as the first argument
> 
> items = list(items) 
> 
> makes of copy of the list, but it will also convert an arbitrary iterable to 
> a list. That's why I can pass the range object.



i really appreciate your help. it works.


From dieter at handshake.de  Wed Dec 14 03:14:27 2016
From: dieter at handshake.de (dieter)
Date: Wed, 14 Dec 2016 09:14:27 +0100
Subject: Name mangling vs qualified access to class attributes
References: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>
Message-ID: <874m276i70.fsf@handshake.de>

paolieri at gmail.com writes:

> The official Python tutorial at
>
> https://docs.python.org/3/tutorial/classes.html#private-variables
>
> says that "name mangling is helpful for letting subclasses override methods without breaking intraclass method calls" and makes an interesting example:
>
> class Mapping:
>     def __init__(self, iterable):
>         self.items_list = []
>         self.__update(iterable)
>
>     def update(self, iterable):
>         for item in iterable:
>             self.items_list.append(item)
>
>     __update = update   # private copy of original update() method
>
> class MappingSubclass(Mapping):
>
>     def update(self, keys, values):
>         # provides new signature for update()
>         # but does not break __init__()
>         for item in zip(keys, values):
>             self.items_list.append(item)
>
>
> It seems to me that, in this example, one could just have:
>
> class Mapping:
>     def __init__(self, iterable):
>         self.items_list = []
>         Mapping.update(self, iterable)
>
>     def update(self, iterable):
>         for item in iterable:
>             self.items_list.append(item)
>
> and avoid copying 'Mapping.update' into 'Mapping.__update'. More generally, any time one needs to "let subclasses override methods without breaking intraclass method calls" (the goal stated in the tutorial), using qualified access to class attributes/methods should suffice.
>
> Am I missing something? Is 'self.__update(iterable)' in 'Mapping.__init__' preferable to 'Mapping.update(self, iterable)'?
>
> I think that, instead, name mangling is helpful to avoid accidental overrides of methods/attributes by the *current* class (rather than its subclasses). Given the way that C3 linearization works, you can't know in advance who will follow your class A in B.__mro__ when B extends A. Name mangling allows you to avoid overriding methods/attributes of classes that might follow.
>
> Any thoughts?

You can do that indeed for class level attributes (such as methods);
you cannot do it for instance level attributes (e.g. holding instance specific
values).

>From my point of view, "__" name mangling is particularly interesting
for mixin classes (i.e. classes implementing a single feature
and being designed to be used in deriving classes combining the necessary
features by deriving from all features classes required).
Those classes are typically combined with other classes - with a
corresponding risk of name clashes. Therefore, the above name mangling
is helpful to reduce that risk for private attributes (whether methods
or data attributes).



From steve+comp.lang.python at pearwood.info  Wed Dec 14 04:38:44 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 14 Dec 2016 20:38:44 +1100
Subject: Is there a way to insert hooks into a native dictionary type to
 see when a query arrives and what's looked up?
References: <o2qno3$oqp$1@dont-email.me>
Message-ID: <58511326$0$2775$c3e8da3$76491128@news.astraweb.com>

On Wednesday 14 December 2016 17:11, Veek M wrote:

> I know that with user classes one can define getattr, setattr to handle
> dictionary lookup. Is there a way to hook into the native dict() type
> and see in real time what's being queried.

Not easily, and maybe not at all.

There are two obvious ways to do this:

(1) monkey-patch the object's __dict__, and the class __dict__.

Unfortunately, Python doesn't support monkey-patching built-ins.

https://en.wikipedia.org/wiki/Monkey_patch

Or perhaps I should say, *fortunately* Python doesn't support it.

http://www.virtuouscode.com/2008/02/23/why-monkeypatching-is-destroying-ruby/

(2) Alternatively, you could make a dict subclass, and replace the class and 
instance __dict__ with your own.

Unfortunately, you cannot replace the __dict__ of a class:

py> class X:  # the class you want to hook into
...     pass
... 
py> class MyDict(dict):  # my custom dict
...     def __getitem__(self, key):
...             print(key)
...             return super().__getitem__(key)
... 
py> d = MyDict()
py> d.update(X.__dict__)
py> X.__dict__ = d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: attribute '__dict__' of 'type' objects is not writable


You can replace the instance dict, but Python won't call your __getitem__ 
method:

py> instance = X()
py> instance.__dict__ = MyDict()
py> instance.a = 999
py> instance.a
999

So the short answer is, No.

You might be able to create a completely new metaclass that supports this, but 
it would be a lot of work, and I'm not even sure that it would be successful.



> I wanted to check if when one does:
> 
> x.sin()
> 
> if the x.__dict__ was queried or if the Foo.__dict__ was queried..

The easiest way to do that is something like this:


py> class Test:
...     def sin(self):
...             return 999
... 
py> x = Test()
py> x.sin
<bound method Test.sin of <__main__.Test object at 0xb6fc3a4c>>
py> x.sin()
999
py> x.sin = "surprise!"
py> x.sin
'surprise!'



So now you know: an instance attribute will shadow the class attribute.

(Actually, that's not *completely* true. It depends on whether x.sin is a 
descriptor or not, and if so, what kind of descriptor.)


-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From p.f.moore at gmail.com  Wed Dec 14 06:43:44 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 14 Dec 2016 03:43:44 -0800 (PST)
Subject: Parsing a potentially corrupted file
Message-ID: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>

I'm looking for a reasonably "clean" way to parse a log file that potentially has incomplete records in it.

The basic structure of the file is a set of multi-line records. Each record starts with a series of fields delimited by [...] (the first of which is always a date), optionally separated by whitespace. Then there's a trailing "free text" field, optionally followed by a multi-line field delimited by [[...]]

So, example records might be

[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here

(a record delimited by the end of the line)

or 

[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here [[Additional
data, potentially multiple lines

including blank lines
goes here
]]

The terminating ]] is on a line of its own.

This is a messy format to parse, but it's manageable. However, there's a catch. Because the logging software involved is broken, I can occasionally get a log record prematurely terminated with a new record starting mid-stream. So something like the following:

[2016-11-30T20:04:08.000+00:00] [Component] [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here

I'm struggling to find a "clean" way to parse this. I've managed a clumsy approach, by splitting the file contents on the pattern [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case where this gets truncated) and then treating each entry as a record and parsing it individually. But the resulting code isn't exactly maintainable, and I'm looking for something cleaner.

Does anyone have any suggestions for a good way to parse this data?

Thanks,
Paul


From rosuav at gmail.com  Wed Dec 14 07:57:03 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 14 Dec 2016 23:57:03 +1100
Subject: Parsing a potentially corrupted file
In-Reply-To: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
References: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
Message-ID: <CAPTjJmojq+=EO5=r4GwY0YpD+8XQmrg-Wo_PxOr2N366UB0ByA@mail.gmail.com>

On Wed, Dec 14, 2016 at 10:43 PM, Paul  Moore <p.f.moore at gmail.com> wrote:
> This is a messy format to parse, but it's manageable. However, there's a catch. Because the logging software involved is broken, I can occasionally get a log record prematurely terminated with a new record starting mid-stream. So something like the following:
>
> [2016-11-30T20:04:08.000+00:00] [Component] [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here
>
> I'm struggling to find a "clean" way to parse this. I've managed a clumsy approach, by splitting the file contents on the pattern [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case where this gets truncated) and then treating each entry as a record and parsing it individually. But the resulting code isn't exactly maintainable, and I'm looking for something cleaner.
>

Is the "[Component]" section something you could verify? (That is - is
there a known list of components?) If so, I would include that as a
secondary check. Ditto anything else you can check (I'm guessing the
[level] is one of a small set of values too.) The logic would be
something like this:

Read line from file.
Verify line as a potential record:
    Assert that line begins with timestamp.
    Verify as many fields as possible (component, level, etc)
    Search line for additional timestamp.
    If additional timestamp found:
        Recurse. If verification fails, assume we didn't really have a
corrupted line.
        (Process partial line? Or discard?)
    If "[[" in line:
        Until line is "]]":
            Read line from file, append to description
            If timestamp found:
                Recurse. If verification succeeds, break out of loop.

 Unfortunately it's still not really clean; but that's the nature of
working with messy data. Coping with ambiguity is *hard*.

ChrisA


From alister.ware at ntlworld.com  Wed Dec 14 08:38:38 2016
From: alister.ware at ntlworld.com (alister)
Date: Wed, 14 Dec 2016 13:38:38 GMT
Subject: Parsing a potentially corrupted file
References: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
Message-ID: <yXb4A.440908$IJ6.366346@fx46.am4>

On Wed, 14 Dec 2016 03:43:44 -0800, Paul  Moore wrote:

> I'm looking for a reasonably "clean" way to parse a log file that
> potentially has incomplete records in it.
> 
> The basic structure of the file is a set of multi-line records. Each
> record starts with a series of fields delimited by [...] (the first of
> which is always a date), optionally separated by whitespace. Then
> there's a trailing "free text" field, optionally followed by a
> multi-line field delimited by [[...]]
> 
> So, example records might be
> 
> [2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id]
> Description of the issue goes here
> 
> (a record delimited by the end of the line)
> 
> or
> 
> [2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id]
> Description of the issue goes here [[Additional data, potentially
> multiple lines
> 
> including blank lines goes here ]]
> 
> The terminating ]] is on a line of its own.
> 
> This is a messy format to parse, but it's manageable. However, there's a
> catch. Because the logging software involved is broken, I can
> occasionally get a log record prematurely terminated with a new record
> starting mid-stream. So something like the following:
> 
> [2016-11-30T20:04:08.000+00:00] [Component]
> [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id]
> Description of the issue goes here
> 
> I'm struggling to find a "clean" way to parse this. I've managed a
> clumsy approach, by splitting the file contents on the pattern
> [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case
> where this gets truncated) and then treating each entry as a record and
> parsing it individually. But the resulting code isn't exactly
> maintainable, and I'm looking for something cleaner.
> 
> Does anyone have any suggestions for a good way to parse this data?
> 
> Thanks,
> Paul

1st question do you (or anyone you can contact) have any control over the 
logging application?

if so the best approach would be to get the log file output fixed.

if not then you will probably be stuck with a messy solution :-(



-- 
Sin has many tools, but a lie is the handle which fits them all.


From p.f.moore at gmail.com  Wed Dec 14 09:07:27 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 14 Dec 2016 06:07:27 -0800 (PST)
Subject: Parsing a potentially corrupted file
In-Reply-To: <mailman.70.1481720231.2322.python-list@python.org>
References: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
 <CAPTjJmojq+=EO5=r4GwY0YpD+8XQmrg-Wo_PxOr2N366UB0ByA@mail.gmail.com>
 <mailman.70.1481720231.2322.python-list@python.org>
Message-ID: <a3692efb-75d8-4160-aa74-27b47adc7e37@googlegroups.com>

On Wednesday, 14 December 2016 12:57:23 UTC, Chris Angelico  wrote:
> Is the "[Component]" section something you could verify? (That is - is
> there a known list of components?) If so, I would include that as a
> secondary check. Ditto anything else you can check (I'm guessing the
> [level] is one of a small set of values too.)

Possibly, although this is to analyze the structure of a basically undocumented log format. So if I validate too tightly, I end up just checking my assumptions rather than checking the data :-(

> The logic would be
> something like this:
> 
> Read line from file.
> Verify line as a potential record:
>     Assert that line begins with timestamp.
>     Verify as many fields as possible (component, level, etc)
>     Search line for additional timestamp.
>     If additional timestamp found:
>         Recurse. If verification fails, assume we didn't really have a
> corrupted line.
>         (Process partial line? Or discard?)
>     If "[[" in line:
>         Until line is "]]":
>             Read line from file, append to description
>             If timestamp found:
>                 Recurse. If verification succeeds, break out of loop.
> 
>  Unfortunately it's still not really clean; but that's the nature of
> working with messy data. Coping with ambiguity is *hard*.

Yeah, that's essentially what I have now. As I say, it's working but nobody could really love it. But you're right, it's more the fault of the data than of the code.

One thought I had, which I might try, is to go with the timestamp as the one assumption I make of the data, and read the file in as, in effect, a text stream, spitting out a record every time I see something matching a the [timestamp] pattern. Then parse record by record. Truncated records should either be obvious (because the delimited fields have start and end markers, so unmatched markers = truncated record) or acceptable (because undelimited fields are free text). I'm OK with ignoring the possibility that the free text contains something that looks like a timestamp.

The only problem with this approach is that I have more data than I'd really like to read into memory all at once, so I'd need to do some sort of streamed match/split processing. But thinking about it, that sounds like the sort of job a series of chained generators could manage. Maybe I'll look at that approach...

Paul


From nick.a.sarbicki at gmail.com  Wed Dec 14 09:08:56 2016
From: nick.a.sarbicki at gmail.com (Nick Sarbicki)
Date: Wed, 14 Dec 2016 14:08:56 +0000
Subject: Wrong release date in 3.6 whats new docs?
Message-ID: <CAGuvt90TLmnZm_wnUTeiykSngo-UrMtesfsqt_ju=W-h=xCfjA@mail.gmail.com>

Afternoon everyone.

Might be missing something obvious but the 3.6 What's New docs point to the
release date being the 12th.

https://docs.python.org/3.6/whatsnew/3.6.html#what-s-new-in-python-3-6

I got the team excited about Friday's release so that caused some confusion
here.

Guessing it's a typo?


From steve+python at pearwood.info  Wed Dec 14 09:12:10 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 15 Dec 2016 01:12:10 +1100
Subject: Name mangling vs qualified access to class attributes
References: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>
Message-ID: <5851533b$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Wed, 14 Dec 2016 07:27 am, paolieri at gmail.com wrote:

> The official Python tutorial at
> 
> https://docs.python.org/3/tutorial/classes.html#private-variables
> 
> says that "name mangling is helpful for letting subclasses override
> methods without breaking intraclass method calls" and makes an interesting
> example:
> 
> class Mapping:
>     def __init__(self, iterable):
>         self.items_list = []
>         self.__update(iterable)
> 
>     def update(self, iterable):
>         for item in iterable:
>             self.items_list.append(item)
> 
>     __update = update   # private copy of original update() method
> 
> class MappingSubclass(Mapping):
> 
>     def update(self, keys, values):
>         # provides new signature for update()
>         # but does not break __init__()
>         for item in zip(keys, values):
>             self.items_list.append(item)
> 
> 
> It seems to me that, in this example, one could just have:
> 
> class Mapping:
>     def __init__(self, iterable):
>         self.items_list = []
>         Mapping.update(self, iterable)
> 
>     def update(self, iterable):
>         for item in iterable:
>             self.items_list.append(item)
> 
> and avoid copying 'Mapping.update' into 'Mapping.__update'. 

Perhaps.

But remember that copying Mapping.update in this way is very cheap: it's
only a new reference (e.g. a copy of a pointer), it doesn't have to copy
the entire function object.

The differences between:

    Mapping.update(self, iterable)

and

    self.__update(iterable)

are very subtle and (as far as I can see) only matter in some fairly hairy
situations. Thanks to name mangling, the second is equivalent to:

    self._Mapping__update(iterable)

which gives subclasses the opportunity to override it, if they dare. They
probably shouldn't, because it is a private method, but it you really,
really need to, you can.

A more exotic difference is that the first example looks directly at the
class, while the second checks for an instance attribute first, giving the
instance the opportunity to shadow _Mapping__update.

One last subtle difference: the second version will work even if you bind
another object to Mapping:

    class Mapping: ...

    instance = Mapping()  # create instance
    Mapping = None  # rebind the name to something else
    d = type(instance)(iterable)  # create a new instance

In this (admittedly exotic) situation Raymond Hettinger's code with
self.__update will continue to work perfectly, while your alternative with
Mapping.update will fail.


I don't know if Raymond has an objective reason for preferring one over the
other, or if it is just a matter of personal taste. If you have a Twitter
account, perhaps you could ask him to comment?

https://twitter.com/raymondh



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From random832 at fastmail.com  Wed Dec 14 09:28:50 2016
From: random832 at fastmail.com (Random832)
Date: Wed, 14 Dec 2016 09:28:50 -0500
Subject: Running python from pty without prompt
In-Reply-To: <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
 <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>
 <mailman.60.1481667900.2322.python-list@python.org>
 <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <1481725730.430333.818813161.20E23856@webmail.messagingengine.com>

On Tue, Dec 13, 2016, at 19:10, Steve D'Aprano wrote:
> Can you show a simple demonstration of what you are doing?
> 
> I'm having difficulty following this thread because I don't know
> what "script run on a tty" means.

The question is literally about the input/script being the tty and not
redirected from any other file, which causes an interactive prompt in
CPython, but does not do so in some other languages. I don't understand
what part of this you're not getting.

> I thought that with the exception of scripts run from cron, any time you
> run a script *or* in interactive mode, there is an associated tty. Am I
> wrong?


From random832 at fastmail.com  Wed Dec 14 09:30:05 2016
From: random832 at fastmail.com (Random832)
Date: Wed, 14 Dec 2016 09:30:05 -0500
Subject: Running python from pty without prompt
In-Reply-To: <58509055$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <mailman.53.1481651778.2322.python-list@python.org>
 <58509055$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <1481725805.430519.818814721.21CC488D@webmail.messagingengine.com>

On Tue, Dec 13, 2016, at 19:20, Steve D'Aprano wrote:
> sys.flags.interactive will tell you whether or not your script was
> launched
> with the -i flag.
> 
> hasattr(sys, 'ps1') or hasattr(sys, 'ps2') will tell you if you are
> running
> in the REPL (interactive interpreter). The ps1 and ps2 variables aren't
> defined in non-interactive mode.

There's no way to *tell python to* run in non-interactive mode without
using a file other than the tty as the script. It's not a matter of
finding out from within python whether it's in interactive note, it's a
matter of python finding out whether the user *wants* it to run in
interactive mode.


From gordon at panix.com  Wed Dec 14 10:59:13 2016
From: gordon at panix.com (John Gordon)
Date: Wed, 14 Dec 2016 15:59:13 +0000 (UTC)
Subject: OT - "Soft" ESC key on the new MacBook Pro
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>	<CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>	<CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>	<CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>	<CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>	<CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>	<CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>	<CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>	<CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>	<CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>	<CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>	<CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>	<mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com> <ebbvoaF750lU1@mid.individual.net>
Message-ID: <o2rq8h$de$2@reader1.panix.com>

In <ebbvoaF750lU1 at mid.individual.net> Gregory Ewing <greg.ewing at canterbury.ac.nz> writes:

> Once you're in the clutches of Apple, there is no Escape.

Ha!

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From ned at nedbatchelder.com  Wed Dec 14 11:23:53 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Wed, 14 Dec 2016 08:23:53 -0800 (PST)
Subject: Wrong release date in 3.6 whats new docs?
In-Reply-To: <mailman.71.1481724551.2322.python-list@python.org>
References: <CAGuvt90TLmnZm_wnUTeiykSngo-UrMtesfsqt_ju=W-h=xCfjA@mail.gmail.com>
 <mailman.71.1481724551.2322.python-list@python.org>
Message-ID: <273775a4-8bab-4de0-88f2-a5656e1e83b9@googlegroups.com>

On Wednesday, December 14, 2016 at 9:09:22 AM UTC-5, Nick Sarbicki wrote:
> Afternoon everyone.
> 
> Might be missing something obvious but the 3.6 What's New docs point to the
> release date being the 12th.
> 
> https://docs.python.org/3.6/whatsnew/3.6.html#what-s-new-in-python-3-6
> 
> I got the team excited about Friday's release so that caused some confusion
> here.
> 
> Guessing it's a typo?

3.6 hasn't been released yet. I think the 12th was the original target date.
3.6.0rc1 was the latest version on Dec 6th, but a problem was discovered
that means an rc2 will be needed.

--Ned.


From torriem at gmail.com  Wed Dec 14 11:57:30 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 14 Dec 2016 09:57:30 -0700
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <ebbvoaF750lU1@mid.individual.net>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com> <ebbvoaF750lU1@mid.individual.net>
Message-ID: <156532a4-2a03-df72-ced8-80aff232d95f@gmail.com>

On 12/13/2016 09:22 PM, Gregory Ewing wrote:
> Paul Rubin wrote:
>> First it was the hipster Mac users
>> with the Beatnik black berets and turtlenecks, and now this. <wink>
> 
> Once you're in the clutches of Apple, there is no Escape.

That's so not true! I've escaped dozens of times! ;)



From none at invalid.com  Wed Dec 14 12:11:51 2016
From: none at invalid.com (mm0fmf)
Date: Wed, 14 Dec 2016 17:11:51 +0000
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <8760mn2pyi.fsf@nightsong.com>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com>
Message-ID: <o2ruea$m1g$1@dont-email.me>

On 14/12/2016 02:40, Paul Rubin wrote:
> Skip Montanaro <skip.montanaro at gmail.com> writes:
>> Does the lack of a physical ESC key create problems for people, especially
>> Emacs users?
>
> Not a Mac user and I rarely use ESC instead of ALT while editing with
> Emacs on a local computer, but when editing remotely I do have to use
> ESC because the Gnome terminal emulator steals a few ALTed keys.  Maybe
> there is a way to stop that behaviour but it didn't occur to me til just
> now.  Hmm.
>
> Meanwhile the concept of a computer with "no escape" just shows Apple
> getting deeper into existentialism.  First it was the hipster Mac users
> with the Beatnik black berets and turtlenecks, and now this. <wink>
>

If you need a full time ESC key then you are just "typing it wrong" as 
Steve Jobs would say if he wasn't dead.



From jon+usenet at unequivocal.eu  Wed Dec 14 12:27:15 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Wed, 14 Dec 2016 17:27:15 -0000 (UTC)
Subject: OT - "Soft" ESC key on the new MacBook Pro
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com> <o2ruea$m1g$1@dont-email.me>
Message-ID: <slrno530a6.298.jon+usenet@sable.unequivocal.eu>

On 2016-12-14, mm0fmf <none at invalid.com> wrote:
> On 14/12/2016 02:40, Paul Rubin wrote:
>> Skip Montanaro <skip.montanaro at gmail.com> writes:
>>> Does the lack of a physical ESC key create problems for people, especially
>>> Emacs users?
>>
>> Not a Mac user and I rarely use ESC instead of ALT while editing with
>> Emacs on a local computer, but when editing remotely I do have to use
>> ESC because the Gnome terminal emulator steals a few ALTed keys.  Maybe
>> there is a way to stop that behaviour but it didn't occur to me til just
>> now.  Hmm.
>>
>> Meanwhile the concept of a computer with "no escape" just shows Apple
>> getting deeper into existentialism.  First it was the hipster Mac users
>> with the Beatnik black berets and turtlenecks, and now this. <wink>
>
> If you need a full time ESC key then you are just "typing it wrong" as 
> Steve Jobs would say if he wasn't dead.

With respect to pressing ESC in Emacs or Vim, you literally don't need
a "full time" ESC key, you only need one when the keyboard input focus
is on a terminal, Emacs or Vim window - and I should imagine that the
ESC key is indeed present whenever that's the case.


From pkpearson at nowhere.invalid  Wed Dec 14 12:40:51 2016
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 14 Dec 2016 17:40:51 GMT
Subject: OT - "Soft" ESC key on the new MacBook Pro
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
Message-ID: <ebdeh2FhsjbU1@mid.individual.net>

On Tue, 13 Dec 2016 19:06:45 -0600, Skip Montanaro wrote:
> I know this isn't a Python-specific question, but 
[snip]
> Yes, I know I can use C-[ or the Alt key instead of ESC.

I know this isn't the sort of answer you wanted, but . . . 

Train your fingers to use C-[.  I did, decades ago, because the darn
escape key kept changing places from one keyboard to the next.  Combined
with the ease with which one can remap the CTRL key to a familiar place,
C-[ has been a blessing.

-- 
To email me, substitute nowhere->runbox, invalid->com.


From skip.montanaro at gmail.com  Wed Dec 14 12:46:47 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 14 Dec 2016 11:46:47 -0600
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <o2ruea$m1g$1@dont-email.me>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <8760mn2pyi.fsf@nightsong.com> <o2ruea$m1g$1@dont-email.me>
Message-ID: <CANc-5Uz2dUeP6hX-FyaA5KZTnHomH2edp9WtbFPYFbEe8ZsTKQ@mail.gmail.com>

> If you need a full time ESC key then you are just "typing it wrong" as Steve Jobs
> would say if he wasn't dead.

Shouldn't the use of ESC, C-[, Alt, or some other mapped key be
treated as a valid personal preference?

I've been using some variety of Emacs (there have been many) since the
early 80s on many different operating systems, with keyboards (or
terminals - think VT100, VT52, or ADM3a) of all types. I think when I
first started using Gosmacs on a VMS machine, ESC was the prefix key,
and Alt wasn't used.

Over the years, I have remapped the backtick and Caps Lock, keys, used
C-[ or Alt, none of which I find suitable, either because I don't wind
up with the same setup across multiple machines (what's the Windows
equivalent of xmodmap?), the darn thing moves around (Sun and PC101
keyboards were always different), is hidden (Alt keys are always
hiding somewhere under my left or right hands), or actually use those
remapped keys for useful stuff already. By-in-large, the ESC key has
remained in the same place on all the keyboards I've ever used. I
trust that a soft ESC key on the Touch Bar will probably be in just
about the right place. My fingers know where the ESC key is without
thinking. I use multiple platforms from time-to-time (typing right now
on a Dell keyboard connected to a Windows machine running Remote
Desktop to connect to a DMZ-hosted Windows machine). My current aging
MBP has a physical ESC key, as does the Apple keyboard attached to my
wife's iMac (which I sometimes use). I'm specifically interested in
how the lack of a physical ESC key affects people who do are used to
the real deal.

If nobody has any experience because the Touch-Bar-equipped MBPs are
too new, that's fine. I thought some people would have purchased it by
now.

Skip


From skip.montanaro at gmail.com  Wed Dec 14 12:50:30 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 14 Dec 2016 11:50:30 -0600
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <ebdeh2FhsjbU1@mid.individual.net>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
Message-ID: <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>

On Wed, Dec 14, 2016 at 11:40 AM, Peter Pearson
<pkpearson at nowhere.invalid> wrote:
> Train your fingers to use C-[.

As I recall, the location of the Ctrl key was one of the differences
between Sun and PC101 keyboards. Doesn't matter so much now, as Sun
has gone the way of the dodo, but it moved around more for me than ESC
over the years.

S


From rosuav at gmail.com  Wed Dec 14 13:44:57 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 15 Dec 2016 05:44:57 +1100
Subject: Recipe request: asyncio "spin off coroutine"
Message-ID: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>

When you work with threads, it's pretty straight-forward to spin off
another thread: you start it. Your current thread continues, and the
other one runs in parallel.

What's the most straight-forward way to do this in asyncio? I know
this has been asked before, but threads keep devolving, and I can't
find back the one I want :| This seems like a recipe that needs to be
in the docs somewhere. Consider:

async def parallel():
    print("Doing stuff in parallel")
    await asyncio.sleep(1)
    print("Done stuff in parallel")

async def do_stuff():
    print("Doing stuff")
    asyncio.spin_off(parallel()) # ???
    print("Doing more stuff")
    await asyncio.sleep(2)
    print("Done doing stuff")

Whenever do_stuff gets called, I want it to run to completion in two
seconds, AND the parallel process should run to completion during that
time.

What code should go on the "???" line to accomplish this?

ChrisA


From marko at pacujo.net  Wed Dec 14 14:27:49 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 14 Dec 2016 21:27:49 +0200
Subject: Recipe request: asyncio "spin off coroutine"
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
Message-ID: <87k2b2jop6.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

>     asyncio.spin_off(parallel()) # ???
>
> [...]
>
> What code should go on the "???" line to accomplish this?

asyncio.ensure_future(parallel())


Marko


From python at mrabarnett.plus.com  Wed Dec 14 14:39:40 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 14 Dec 2016 19:39:40 +0000
Subject: Parsing a potentially corrupted file
In-Reply-To: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
References: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
Message-ID: <520dc397-1ec5-ec69-3cfb-366c7448376b@mrabarnett.plus.com>

On 2016-12-14 11:43, Paul Moore wrote:
> I'm looking for a reasonably "clean" way to parse a log file that potentially has incomplete records in it.
>
> The basic structure of the file is a set of multi-line records. Each record starts with a series of fields delimited by [...] (the first of which is always a date), optionally separated by whitespace. Then there's a trailing "free text" field, optionally followed by a multi-line field delimited by [[...]]
>
> So, example records might be
>
> [2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here
>
> (a record delimited by the end of the line)
>
> or
>
> [2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here [[Additional
> data, potentially multiple lines
>
> including blank lines
> goes here
> ]]
>
> The terminating ]] is on a line of its own.
>
> This is a messy format to parse, but it's manageable. However, there's a catch. Because the logging software involved is broken, I can occasionally get a log record prematurely terminated with a new record starting mid-stream. So something like the following:
>
> [2016-11-30T20:04:08.000+00:00] [Component] [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here
>
> I'm struggling to find a "clean" way to parse this. I've managed a clumsy approach, by splitting the file contents on the pattern [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case where this gets truncated) and then treating each entry as a record and parsing it individually. But the resulting code isn't exactly maintainable, and I'm looking for something cleaner.
>
> Does anyone have any suggestions for a good way to parse this data?
>
I think I'd do something like this:

while have_more(input):
     # At the start of a record.
     timestamp = parse_timestamp(input)

     fields = []
     description = None
     additional = None

     try:
         for i in range(5):
             # A field shouldn't contain a '[', so if it sees one one, it'll
             # push it back and return True for truncated.
             field, truncated = parse_field(input)
             fields.append(fields)

             if truncated:
                 raise TruncatedError()

         # The description shouldn't contain a timestamp, but if it 
does, it'll
         # push it back from that point and return True for truncated.
         description, truncated = parse_description(input)

         if truncated:
             raise TruncatedError()

         # The additional information shouldn't contain a timestamp, but 
if it
         # does, it'll push it back from that point and return True for
         # truncated.
         additional, truncated = parse_additional_information(input)

         if truncated:
             raise TruncatedError()
     except TruncatedError:
         process_record(timestamp, fields, description, additional, 
truncated=True)
     else:
         process_record(timestamp, fields, description, additional)



From rosuav at gmail.com  Wed Dec 14 14:53:31 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 15 Dec 2016 06:53:31 +1100
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <87k2b2jop6.fsf@elektro.pacujo.net>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
 <87k2b2jop6.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>

On Thu, Dec 15, 2016 at 6:27 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Chris Angelico <rosuav at gmail.com>:
>
>>     asyncio.spin_off(parallel()) # ???
>>
>> [...]
>>
>> What code should go on the "???" line to accomplish this?
>
> asyncio.ensure_future(parallel())

Hmm. I tried that but it didn't work in the full program (it hung the
calling coroutine until completion). But it does work in the toy
example I posted here. Weird. Must be something else that's wrong,
then. I'll keep poking around, thanks.

ChrisA


From marko at pacujo.net  Wed Dec 14 15:25:19 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 14 Dec 2016 22:25:19 +0200
Subject: Recipe request: asyncio "spin off coroutine"
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
 <87k2b2jop6.fsf@elektro.pacujo.net>
 <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>
 <mailman.79.1481745220.2322.python-list@python.org>
Message-ID: <87a8byjm1c.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Thu, Dec 15, 2016 at 6:27 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Chris Angelico <rosuav at gmail.com>:
>>
>>>     asyncio.spin_off(parallel()) # ???
>>>
>>> [...]
>>>
>>> What code should go on the "???" line to accomplish this?
>>
>> asyncio.ensure_future(parallel())
>
> Hmm. I tried that but it didn't work in the full program (it hung the
> calling coroutine until completion). But it does work in the toy
> example I posted here. Weird. Must be something else that's wrong,
> then. I'll keep poking around, thanks.

What version of Python are you running?

   Changed in version 3.5.1: The function accepts any awaitable object.

   <URL: https://docs.python.org/3/library/asyncio-task.html#asyncio.ensu
   re_future>


Marko


From rosuav at gmail.com  Wed Dec 14 15:32:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 15 Dec 2016 07:32:28 +1100
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <87a8byjm1c.fsf@elektro.pacujo.net>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
 <87k2b2jop6.fsf@elektro.pacujo.net>
 <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>
 <mailman.79.1481745220.2322.python-list@python.org>
 <87a8byjm1c.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmr9DhW+z1gya+zx_og4sqSymQyyGW_hm+w0VFq5tRwGdA@mail.gmail.com>

On Thu, Dec 15, 2016 at 7:25 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> What version of Python are you running?
>
>    Changed in version 3.5.1: The function accepts any awaitable object.
>

3.7 :)

ChrisA


From ian.g.kelly at gmail.com  Wed Dec 14 15:53:50 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 14 Dec 2016 13:53:50 -0700
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
 <87k2b2jop6.fsf@elektro.pacujo.net>
 <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>
Message-ID: <CALwzidm-WTMTVP36=fJHe9fmy3YAT0YHGXhpVMueX6475fkH+g@mail.gmail.com>

On Wed, Dec 14, 2016 at 12:53 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Thu, Dec 15, 2016 at 6:27 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Chris Angelico <rosuav at gmail.com>:
>>
>>>     asyncio.spin_off(parallel()) # ???
>>>
>>> [...]
>>>
>>> What code should go on the "???" line to accomplish this?
>>
>> asyncio.ensure_future(parallel())
>
> Hmm. I tried that but it didn't work in the full program (it hung the
> calling coroutine until completion). But it does work in the toy
> example I posted here. Weird. Must be something else that's wrong,
> then. I'll keep poking around, thanks.

Did you just use "asyncio.ensure_future(parallel())" in the full
program or did you throw in an await by mistake?


From rosuav at gmail.com  Wed Dec 14 16:05:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 15 Dec 2016 08:05:58 +1100
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <CALwzidm-WTMTVP36=fJHe9fmy3YAT0YHGXhpVMueX6475fkH+g@mail.gmail.com>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <mailman.77.1481741105.2322.python-list@python.org>
 <87k2b2jop6.fsf@elektro.pacujo.net>
 <CAPTjJmpEP+5nsA5h=Ov64KWXZrmgecGFha_QEtKzfhg1a7-FYQ@mail.gmail.com>
 <CALwzidm-WTMTVP36=fJHe9fmy3YAT0YHGXhpVMueX6475fkH+g@mail.gmail.com>
Message-ID: <CAPTjJmoQ4ndWAJ-qQjXK4n22g4Eo4g=WJRYE5cGNKXEf8KXwJQ@mail.gmail.com>

On Thu, Dec 15, 2016 at 7:53 AM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> On Wed, Dec 14, 2016 at 12:53 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Thu, Dec 15, 2016 at 6:27 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>>> Chris Angelico <rosuav at gmail.com>:
>>>
>>>>     asyncio.spin_off(parallel()) # ???
>>>>
>>>> [...]
>>>>
>>>> What code should go on the "???" line to accomplish this?
>>>
>>> asyncio.ensure_future(parallel())
>>
>> Hmm. I tried that but it didn't work in the full program (it hung the
>> calling coroutine until completion). But it does work in the toy
>> example I posted here. Weird. Must be something else that's wrong,
>> then. I'll keep poking around, thanks.
>
> Did you just use "asyncio.ensure_future(parallel())" in the full
> program or did you throw in an await by mistake?

I didn't await that, no. Have just pinned down the problem, and it's a
total facepalm moment: the secondary task had gotten stuck in an
infinite loop with no await points in it. So technically they _would_
both have been scheduled when I did it as per Marko's suggestion, but
I couldn't tell. Whoooooooops. Sorry about that!

Still might be worth having a simple recipe in the docs though; it
would have saved me the trouble of digging through my "fork" code.

ChrisA


From no.email at nospam.invalid  Wed Dec 14 17:43:53 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Wed, 14 Dec 2016 14:43:53 -0800
Subject: Parsing a potentially corrupted file
References: <5256a86e-af72-429c-8744-f550105274af@googlegroups.com>
Message-ID: <87y3zi168m.fsf@nightsong.com>

Paul  Moore <p.f.moore at gmail.com> writes:
> I'm looking for a reasonably "clean" way to parse a log file that
> potentially has incomplete records in it.

Basically trial and error.  Code something reasonable, run your program
til it crashes on a record that it doesn't know what to do with, add
code to deal with that, rinse and repeat.  I've done this kind of thing
multiple times.  You tend to get exponentially further along with each
run/crash/fix iteration til you get most of everything, though there
might still be an occasional hopeless record that you have to just log.


From breamoreboy at gmail.com  Wed Dec 14 17:53:20 2016
From: breamoreboy at gmail.com (breamoreboy at gmail.com)
Date: Wed, 14 Dec 2016 14:53:20 -0800 (PST)
Subject: Wrong release date in 3.6 whats new docs?
In-Reply-To: <mailman.71.1481724551.2322.python-list@python.org>
References: <CAGuvt90TLmnZm_wnUTeiykSngo-UrMtesfsqt_ju=W-h=xCfjA@mail.gmail.com>
 <mailman.71.1481724551.2322.python-list@python.org>
Message-ID: <876ce65f-b7f0-4e68-9478-046a34c60251@googlegroups.com>

On Wednesday, December 14, 2016 at 2:09:22 PM UTC, Nick Sarbicki wrote:
> Afternoon everyone.
> 
> Might be missing something obvious but the 3.6 What's New docs point to the
> release date being the 12th.
> 
> https://docs.python.org/3.6/whatsnew/3.6.html#what-s-new-in-python-3-6
> 
> I got the team excited about Friday's release so that caused some confusion
> here.
> 
> Guessing it's a typo?

Are you confusing the date on which the what's new was updated with the release schedule here https://www.python.org/dev/peps/pep-0494/ ?

Kindest regards.

Mark Lawrence.


From steve+python at pearwood.info  Wed Dec 14 18:27:08 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 15 Dec 2016 10:27:08 +1100
Subject: Running python from pty without prompt
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <e6c14baa-d220-4bf9-8897-7add0dba5e04@googlegroups.com>
 <f3f4c346-a6ea-6a95-4538-7bd0c0e08d80@gmail.com>
 <1481651287.136388.817796337.07F2714E@webmail.messagingengine.com>
 <c582b581-3475-9126-ed7e-0f6bd79ee904@gmail.com>
 <1481667897.205877.818087297.0B5D617A@webmail.messagingengine.com>
 <mailman.60.1481667900.2322.python-list@python.org>
 <58508e01$0$1604$c3e8da3$5496439d@news.astraweb.com>
 <1481725730.430333.818813161.20E23856@webmail.messagingengine.com>
 <mailman.72.1481725739.2322.python-list@python.org>
Message-ID: <5851d54e$0$1603$c3e8da3$5496439d@news.astraweb.com>

On Thu, 15 Dec 2016 01:28 am, Random832 wrote:

> On Tue, Dec 13, 2016, at 19:10, Steve D'Aprano wrote:
>> Can you show a simple demonstration of what you are doing?
>> 
>> I'm having difficulty following this thread because I don't know
>> what "script run on a tty" means.
> 
> The question is literally about the input/script being the tty and not
> redirected from any other file, which causes an interactive prompt in
> CPython, but does not do so in some other languages. I don't understand
> what part of this you're not getting.

What can I say? Maybe I'm just slow. Or maybe you're falling into the curse
of knowledge:

https://en.wikipedia.org/wiki/Curse_of_knowledge

I'm not the only one having trouble understanding the nature of this
problem -- Michael Torrie has also said "though a tty comes into this
somehow and I'm not clear on that".

What is meant by "the input/script being the tty"? And how does that relate
to the subject line which refers to a pty?

That's why I've asked for a simple example that demonstrates the issue. But
apparently this simple example is so simple that nobody knows how to write
it. I cannot replicate the OP's problem from his description alone, and I
have not seen an example where the Python prompt is shown apart from when
running Python interactively.

So... the input is the tty. I don't know what that means, but I think I know
what it isn't. I'm fairly confident it isn't when you pipe the output of
one process to Python:

# not this
[steve at ando ~]$ echo "import sys; print sys.version" | python
2.7.2 (default, May 18 2012, 18:25:10)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]


And likewise you said it is not when input is *redirected* from a file, so
it probably isn't this:

[steve at ando ~]$ cat myfile
import sys; print sys.version

[steve at ando ~]$ python < myfile
2.7.2 (default, May 18 2012, 18:25:10)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]


and surely I can eliminate passing the file name as a command line argument
(python myfile) as well. So what is left?


Michael Torrie suggests something more or less like this, redirecting stdin
to Python with a here-doc:

[steve at ando ~]$ python << .
> import sys
> print sys.version
> .
2.7.2 (default, May 18 2012, 18:25:10) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]

(Michael's version used EOF rather than a dot.) There's a prompt, but it's
not the Python prompt, it's from the shell. Since you are insisting that
the Python interactive prompt is involved, then surely Michael's example
isn't what you mean either.

So I now have *four* ways of running code in Python that *don't* match the
OP's problem (five if you include the standard REPL) and I'm not closer to
understanding the OP's problem.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From renjithmadhavan at gmail.com  Wed Dec 14 20:56:10 2016
From: renjithmadhavan at gmail.com (renjith madhavan)
Date: Wed, 14 Dec 2016 17:56:10 -0800 (PST)
Subject: Python3, column names from array - numpy or pandas
Message-ID: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>

I have a dataset in the below format.

id	A	B	C	D	E
100	1	0	0	0	0
101	0	1	1	0	0
102	1	0	0	0	0
103	0	0	0	1	1

I would like to convert this into below:
100, A
101, B C
102, A
103, D E

How do I do this ? I tried numpy argsort but I am new to Python and finding this challenging.
Appreciate any help in this.


From PointedEars at web.de  Wed Dec 14 23:14:18 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Thu, 15 Dec 2016 05:14:18 +0100
Subject: Python constructors have particular semantics,
 and =?UTF-8?B?4oCYRm9vLl9faW5pdF9f4oCZ?= doesn't qualify
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
 <mailman.38.1481587045.2322.python-list@python.org>
Message-ID: <2311421.k3LOHGUjKi@PointedEars.de>

Ben Finney wrote:

> Chris Angelico <rosuav at gmail.com> writes:
>> On Tue, Dec 13, 2016 at 10:17 AM, Ben Finney <ben+python at benfinney.id.au>
>> wrote:
>> > If the differences didn't matter I would agree that ?overly
>> > pedantic? is fair. But those differences trip up newcomers. Thinking
>> > of ?Foo.__init__? leads people to wonder where the ?self? attribute
>> > came from ? am I not meant to be constructing it? ? and to attempt
>> > to return that instance. And when the time comes to lean about
>> > ?__new__? the confusion continues, because the newcomer has been
>> > told that something *else* is the constructor, so what's this?
>>
>> In JavaScript [different semantics apply].
>>
>> Ultimately, every language has slightly different semantics [?], so
>> you have to learn that the "constructor" might have slightly different
>> semantics.

Actually, that is an inverse semantic fallacy.  There *is* always *some* 
common semantics to a the same term used in different domains; that is *why* 
it is used across domains (here: across object-oriented programming 
languages).  The common semantics here appears to be, as I substantiated by 
the FOLDOC reference, that a constructor initializes an object/instance.

?construct? does _not_ mean the same as ?create? (make); understood 
literally, it means ?heap together?, ?build? (from Latin ?construo?, from 
?com? ?together? + ?struo? ?I heap up, pile?).  A ?constructor? is then the 
person or (here) the thing that ?heaps together?.  ISTM that it ?heaps 
together? the properties/attributes so that the object can be useful.

This is the same in Python as in several other OOPLs inasfar as that I can 
use super() or the superclass? name to call the parent ?what? ? yes, the 
parent *constructor* of the class, i.e. the superclass? (or in python, one 
of the superclass?) *constructor* from a class? *constructor* (yes, I 
insist; see below) to initialize inherited attributes.

> Please read again the message you extracted that quote from. I already
> said I'm not claiming some other programming language's semantics should
> dictate Python's.
> 
> 
> What I'm saying is that in Python, there *already are* different
> semantics for a constructor, and they don't match the semantics of
> ?Foo.__init__?.

Yes, *you* are saying that.
 
> In Python, a constructor for a class is a class method.

Says who, *you*?

> ?Foo.__new__? is a constructor. ?Foo.__init__? is an instance method, so
> it's not a constructor.

According to <https://docs.python.org/3/tutorial/classes.html#method-objects>, ?Foo.__init__? is _not_ an instance method.  Were it an instance 
method, the following would not happen:

| >>> class Foo:
| ...     def __init__ (self):
| ...         pass
| ...
| >>> Foo.__init__.__self__
| Traceback (most recent call last):
|   File "<stdin>", line 1, in <module>
| AttributeError: 'function' object has no attribute '__self__'

Because ?Instance method objects have attributes, too: m.__self__ is the 
instance object with the method m() [?]?.

?Ex falso quodlibet?, ?wishful thinking?, and ?circular reasoning? come to 
mind here: *You* say that a constructor is a class method in Python, then 
*you* define __init__() to be an instance method, therefore __init__() 
*cannot* qualify anymore as a constructor.  So your wish is fulfilled, *far 
from reality*.

[If one would instantiate Foo, *then* the __init__() method of the instance 
would be an instance method (which is tautological):

| >>> Foo().__init__.__self__
| <__main__.Foo object at 0x7f86d73e8ef0>

]
 
> In Python, a constructor for a class makes the instance where it didn't
> already exist. ?Foo.__new__? is a constructor. ?Foo.__init__? requires
> the instance to already be constructed, so it's not a constructor.

You are also limiting the meaning of ?constructor? in OOP to ?create the 
instance? so that __init__() does not fit your definition of what is a 
constructor in Python in that regard as well.  But as I showed, not only 
does ?constructor? not need to mean that, it can also mean much more than 
that.

So your reasoning is doubly invalid.
 
> I'm showing that Python classes *already have* constructors,

By *your* definition that you have *construed* (no pun intended) to fit your 
argument.

> and ?Foo.__init__? doesn't qualify because it doesn't have the semantics 
> of Python constructors.

By *your* definition.  That does not mean that your definition is correct, 
or that your reasoning is valid.

-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From space.ship.traveller at gmail.com  Wed Dec 14 23:29:31 2016
From: space.ship.traveller at gmail.com (Samuel Williams)
Date: Wed, 14 Dec 2016 20:29:31 -0800 (PST)
Subject: Running python from pty without prompt
In-Reply-To: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
Message-ID: <cc1bc316-a186-4315-980d-46ce4b967151@googlegroups.com>

Here are some examples of different languages:

https://github.com/ioquatix/script-runner/blob/master/examples/python-eot.py


From miki.tebeka at gmail.com  Thu Dec 15 00:04:41 2016
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Wed, 14 Dec 2016 21:04:41 -0800 (PST)
Subject: Python3, column names from array - numpy or pandas
In-Reply-To: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>
References: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>
Message-ID: <0418ef59-1c34-4085-8fcd-469b1dd7ed13@googlegroups.com>

You can do this with pandas:

    import pandas as pd
    from io import StringIO

    io = StringIO('''\
    id        A        B        C        D        E 
    100        1        0        0        0        0 
    101        0        1        1        0        0 
    102        1        0        0        0        0 
    103        0        0        0        1        1 
    ''')

    df = pd.read_csv(io, sep='\s+', index_col='id')
    val = df.apply(lambda row: ' '.join(df.columns[row==1]), axis=1)

Google for "boolean indexing" to see what's going on :)


From torriem at gmail.com  Thu Dec 15 00:08:59 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 14 Dec 2016 22:08:59 -0700
Subject: Running python from pty without prompt
In-Reply-To: <cc1bc316-a186-4315-980d-46ce4b967151@googlegroups.com>
References: <c93debe8-6ddf-4966-a0bb-8d744a1325da@googlegroups.com>
 <cc1bc316-a186-4315-980d-46ce4b967151@googlegroups.com>
Message-ID: <370f7f97-030f-4291-fe6e-046071fc8e03@gmail.com>

On 12/14/2016 09:29 PM, Samuel Williams wrote:
> Here are some examples of different languages:
> 
> https://github.com/ioquatix/script-runner/blob/master/examples/python-eot.py

Okay so it looks like you're just opening a pipe to a subprocess and
feeding it a script and input.  So there's no pty involved here.  Or am
I wrong?

In any case, I think if you made a python wrapper script that could take
the standard in up until the ctrl-d, and then exec() that, what's left
on standard in should be able to feed the exec'd script any input it needs.




From frank at chagford.com  Thu Dec 15 00:39:59 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 15 Dec 2016 07:39:59 +0200
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
Message-ID: <o2tabb$dvk$1@blaine.gmane.org>

"Chris Angelico"  wrote in message 
news:CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA at mail.gmail.com...
>
> When you work with threads, it's pretty straight-forward to spin off
> another thread: you start it. Your current thread continues, and the
> other one runs in parallel.
>
> What's the most straight-forward way to do this in asyncio?
>

You have got the answer - asyncio.ensure_future(...)

I thought I would share something that happened just yesterday, where this 
came to my rescue.

I mentioned in a recent post that I had set up a background task to process 
HTTP requests for a given session, using an asyncio.Queue and an endless 
loop.

I have a session.close() coroutine which has, among other steps -
    await self.request_queue.put(None)  # to stop the loop
    await self.request_queue.join()  # to ensure all requests completed

There are three places in my code that can await session.close() -

    - on closing the program, close all open sessions
    - on detecting that the session is no longer responding, kill the 
session
    - on receiving a message from the client indicating that it had closed

The first two worked fine, but the third one hung. I eventually found that, 
because I was awaiting session.close() from within the request handler, that 
request had not completed, therefore it was hanging on the 'join'.

I was stumped for a while, and then I had an 'aha' moment.

I changed 'await self.close()', to 'asyncio.ensure_future(self.close())'.

Problem solved.

Frank Millman





From marko at pacujo.net  Thu Dec 15 01:32:29 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 15 Dec 2016 08:32:29 +0200
Subject: Recipe request: asyncio "spin off coroutine"
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <o2tabb$dvk$1@blaine.gmane.org>
 <mailman.84.1481780412.2322.python-list@python.org>
Message-ID: <87vaulitxe.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:

> I changed 'await self.close()', to 'asyncio.ensure_future(self.close())'.
>
> Problem solved.

A nice insight.

However, shouldn't somebody somewhere in your code be keeping track of
the returned task?


Marko


From nick.a.sarbicki at gmail.com  Thu Dec 15 01:51:03 2016
From: nick.a.sarbicki at gmail.com (Nick Sarbicki)
Date: Thu, 15 Dec 2016 06:51:03 +0000
Subject: Wrong release date in 3.6 whats new docs?
In-Reply-To: <876ce65f-b7f0-4e68-9478-046a34c60251@googlegroups.com>
References: <CAGuvt90TLmnZm_wnUTeiykSngo-UrMtesfsqt_ju=W-h=xCfjA@mail.gmail.com>
 <mailman.71.1481724551.2322.python-list@python.org>
 <876ce65f-b7f0-4e68-9478-046a34c60251@googlegroups.com>
Message-ID: <CAGuvt93qEx1+D5BNma+dc_g5US=DXWWu3z_cK9d5s5LPjxNyFQ@mail.gmail.com>

I'm aware of the the schedule in the PEP.

But if the date at the top of the What's New page is the last day it was
updated and not the release date then that is what has caused the confusion.

On Wed, 14 Dec 2016, 22:58 , <breamoreboy at gmail.com> wrote:

> On Wednesday, December 14, 2016 at 2:09:22 PM UTC, Nick Sarbicki wrote:
> > Afternoon everyone.
> >
> > Might be missing something obvious but the 3.6 What's New docs point to
> the
> > release date being the 12th.
> >
> > https://docs.python.org/3.6/whatsnew/3.6.html#what-s-new-in-python-3-6
> >
> > I got the team excited about Friday's release so that caused some
> confusion
> > here.
> >
> > Guessing it's a typo?
>
> Are you confusing the date on which the what's new was updated with the
> release schedule here https://www.python.org/dev/peps/pep-0494/ ?
>
> Kindest regards.
>
> Mark Lawrence.
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From frank at chagford.com  Thu Dec 15 02:19:03 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 15 Dec 2016 09:19:03 +0200
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <87vaulitxe.fsf@elektro.pacujo.net>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <o2tabb$dvk$1@blaine.gmane.org>
 <mailman.84.1481780412.2322.python-list@python.org>
 <87vaulitxe.fsf@elektro.pacujo.net>
Message-ID: <o2tg53$pio$1@blaine.gmane.org>

"Marko Rauhamaa"  wrote in message news:87vaulitxe.fsf at elektro.pacujo.net...
>
> "Frank Millman" <frank at chagford.com>:
>
> > I changed 'await self.close()', to 
> > 'asyncio.ensure_future(self.close())'.
> >
> > Problem solved.
>
> A nice insight.
>
> However, shouldn't somebody somewhere in your code be keeping track of
> the returned task?
>

I don't know. What is the worst that could happen?

My way of looking at it is that it is similar to setTimeout() in javascript. 
I am requesting that the enclosed function/coroutine be scheduled for 
execution at the next available opportunity in the event loop. In 
javascript, I don't keep track of it, I just assume that it will be executed 
at some point. Is it not reasonable to do the same here?

Frank




From marko at pacujo.net  Thu Dec 15 02:28:57 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 15 Dec 2016 09:28:57 +0200
Subject: Recipe request: asyncio "spin off coroutine"
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <o2tabb$dvk$1@blaine.gmane.org>
 <mailman.84.1481780412.2322.python-list@python.org>
 <87vaulitxe.fsf@elektro.pacujo.net> <o2tg53$pio$1@blaine.gmane.org>
 <mailman.86.1481786354.2322.python-list@python.org>
Message-ID: <87mvfxirba.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:

> "Marko Rauhamaa"  wrote in message news:87vaulitxe.fsf at elektro.pacujo.net...
>>
>> "Frank Millman" <frank at chagford.com>:
>>
>> > I changed 'await self.close()', to >
>> 'asyncio.ensure_future(self.close())'.
>> >
>> > Problem solved.
>>
>> A nice insight.
>>
>> However, shouldn't somebody somewhere in your code be keeping track of
>> the returned task?
>
> I don't know. What is the worst that could happen?

Only you can tell.

> I just assume that it will be executed at some point. Is it not
> reasonable to do the same here?

It ain't over till the fat lady sings. Things can accumulate, hang
and/or fail in surprising ways.

At the very least you should maintain statistics that reveal the number
of pending closing tasks for troubleshooting when things go south.


Marko


From frank at chagford.com  Thu Dec 15 03:16:00 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 15 Dec 2016 10:16:00 +0200
Subject: Recipe request: asyncio "spin off coroutine"
In-Reply-To: <87mvfxirba.fsf@elektro.pacujo.net>
References: <CAPTjJmoFyJqYw4G_kNo5Sn=ULyjgm=KeXqrQVWUBr87EVbzAFA@mail.gmail.com>
 <o2tabb$dvk$1@blaine.gmane.org>
 <mailman.84.1481780412.2322.python-list@python.org>
 <87vaulitxe.fsf@elektro.pacujo.net> <o2tg53$pio$1@blaine.gmane.org>
 <mailman.86.1481786354.2322.python-list@python.org>
 <87mvfxirba.fsf@elektro.pacujo.net>
Message-ID: <o2tjfs$cb0$1@blaine.gmane.org>

"Marko Rauhamaa"  wrote in message news:87mvfxirba.fsf at elektro.pacujo.net...
>
> It ain't over till the fat lady sings. Things can accumulate, hang
> and/or fail in surprising ways.
>
> At the very least you should maintain statistics that reveal the number
> of pending closing tasks for troubleshooting when things go south.
>

I know that you have vastly more experience in this area than I do, so I 
will take your advice to heart.

Thanks

Frank




From paolieri at gmail.com  Thu Dec 15 05:36:18 2016
From: paolieri at gmail.com (Marco Paolieri)
Date: Thu, 15 Dec 2016 02:36:18 -0800 (PST)
Subject: Name mangling vs qualified access to class attributes
In-Reply-To: <5851533b$0$1608$c3e8da3$5496439d@news.astraweb.com>
References: <b0430bd7-97f1-455c-aeb4-c4ba065eac35@googlegroups.com>
 <5851533b$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <9d9e5934-5c30-4dfd-829d-e1fe1fcb03d5@googlegroups.com>

Hi all,

Thank you for the feedback. So, to recap, reasons to use name mangling
'self.__update(iterable)' rather than qualified access
'Mapping.update(self, iterable)' are:


1. Qualified access stops working when class 'Mapping' is renamed
(at compile-time) or its name is reassigned at runtime.

Honestly, I wouldn't worry about the first case: if I rename a class,
fixing qualified accesses within the class itself is just a matter of
refactoring.  Instead, when another class is assigned to the name
'Mapping' (in the scope where it was defined), qualified access breaks
at runtime, while looking up methods through 'self' still works.


2. Method lookup through 'self' allows subclasses or instances to
override mangled names.

This should be extremely rare. After all, mangled names are meant to
avoid overrides.


For me, the takeaway is that it's always better to invoke methods on
'self', rather than passing 'self' yourself. As Guido said about the
drawbacks of having methods see each other in their scopes:

"m1(self) and self.m1() [would be] equivalent.  That's evil, because
it opens up a meaningless choice between alternative (and the m1(self)
notation is inferior, because it doesn't look in base classes)."

https://mail.python.org/pipermail/python-dev/2000-November/010598.html


Best,
-- Marco


From skybuck2000 at hotmail.com  Thu Dec 15 06:23:43 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Thu, 15 Dec 2016 03:23:43 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
Message-ID: <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>

Hello,

I received a reply from somebody on my ISP newsserver. Apperently his reply is not visible on google groups. I wonder why, maybe it's a banned troll or something, but perhaps not.

Anyway... my ISP has problems accessing their newsserver. They won't offer the service to new customers or changing subscriptions and they lack software to access the server. The server currently has a technical problem. Uploading messages is not possible. Some kind of bug... this has been going on for months. Seems like the bug will be ever lasting.

Just called ISP helpdesk... I forgot to ask them if they could simply reset the server... but with the little information I was giving it seems to be a "remote control problem".

The server is probably remotely controlled and somehow that remote control has been lost... Either password lost, or software is not working anymore... kinda strange story... especially for such a large ISP which should have sufficient technical people to solve this... weird story isn't it... perhaps they just don't want to fix it to drop people of of it....

This kinda sucks for me because using usenet via google newsgroups completely sucks... and is like 1.000.000 times more difficult with this very lacking interface.

But for messages that I consider worth it... like this one, I might keep on going(/presserve? gotta look that up in dictionary) for now... :)

Anyway here is my reply, it might enlighting others as well to this exact problem:


>So now I have the idea to make this program run when my computer is idling
>during the day, it should also be able to store it's progress so that it 
>can
>continue after it was shutdown.
>

"
Periodic snap-shots of the running environment will likely take up a
majority of the run-time.
"

(Since you the only one who has replied so far I will take some time to try 
and answer some of your questions to help you along and perhaps it will be 
usefull to others as well).

Haven't implemented this yet, but I don't see why that would be the case.

It would probably need to store very little. Only 8 indexes, and perhaps 
4x24 entries so that's nothing.

This could be then everything index4 increments for example, or index 3.

Somewhere below in this posting I will give some pseudo code for the indexes 
since it seems you might be struggling with that a little bit or you are 
trying to solve it a different way which is interesting to see but in this 
case I don't think it's better so I will adjust you there, but first

I reply to the rest of your posting so the pseudo code will be somewhere 
below...

"
Especially if your idea is to first generate two lists of the permutations 
of the inputs
"

The permutations were already given by me in my original post. No further 
permutation lists are needed. The permutation table was simple generated 
with some 1234 string with some Delphi code found from the internet.

Generating permutations is not the real issue here, since there is code on 
the internet which already solves that.

"
then generate a list of the "combat" permutations, and you want to be able 
to store-off/reload all
those lists.
"

I don't see why it would be necessary to store any lists at all, except from 
the permutation table as to avoid having to implement a complex permutation 
algorithm. Generating permutations is a topic in itself so I tried to avoid 
that by simply hard coding the small permutation list into the code to side 
step that sub problem.

Generating/storing/loading any kind combination list would either not be 
possible because of lack of memory or would require way too much time.

What should however be stored is the number of victories for each 
permutation.

>(Idea for now is to make it multi threaded and assign a low thread priority

"It's CPU-bound, so using multiple threads for the computation won't be
that effective..."

I disagree, the problem is pretty straight forward and can be split just 
fine by duplicating some simple indexes/data structures.

And should be able to run on each core at nearly 100% efficiency.

The results could be merged together at the very end.

At least in Delphi/native code this will work nicely...

"You add the system overhead of context switches between
the threads, and for common Python, you run into the GIL (so even if you
get subsets of processing assigned to each of multiple processor cores, the
GIL will block all but one from doing any work at any moment -- say you
partitioned it so core0 handles only the set where X begins [1, ...], core
1 handles [2, ...], etc.).
"

Your last sentence made the most sense to me... I don't know what GIL is or 
if Python has any multi threading issues or overhead.

I would assume it would not be to bad concerning overhead and that python 
threads can run pretty efficient as well, if not that would surprise me a 
little bit.

Ofcourse there would only be 2 threads since it's a dual core system. If you 
ment running more than 2 threads then yes there would be context overswitch 
overhead.

But why would anybody want to run more threads than the number of cores ?! 
;) Since it's not necessary to run more threads to divide up the problem.

>so it can run during the day when I use my computer and it's not doing much
>so it can use the reserve computational horse power).
>(I still have to try these "idle/reverse" ideas to see which one works best
>without interrupting my web browsing or music listening too much ;))
>

<snip>


"
Not seeing any code makes it hard to estimate where your slow-downs
would be, however all your mentions of keeping "index" values implies (as I
mention above) that you are trying to build huge tables of permutations
first, then beating each entry against those of the other.
"

I quickly realised that this approach (generating huge lists) won't work 
because out of memory.

So instead each possibility is simply calculated and only results are 
stored.

"In truth, I can't figure out what you consider a combat"

I did not use the term "combat". I do think of them as "attacks".

We could define "combat" as simple "one of the possibilities".

So let's look at a single possibility to explain it to you.

I will simply produce a random index example:

Index1: 7
Index2: 22
Index3: 12
Index4: 10
Index5: 4
Index6: 9
Index7: 11
Index8: 5

Now the task for program is to calculate the outcome of battle.

The program will eventually lookup the permutations from the permutation 
table so let's get those first:

First let's copy & paste the permutation table from original posting:

1,2,3,4    // 1
1,2,4,3    // 2
1,3,2,4    // 3
1,3,4,2    // 4
1,4,2,3    // 5
1,4,3,2    // 6
2,1,3,4    // 7
2,1,4,3    // 8
2,3,1,4    // 9
2,3,4,1    // 10
2,4,1,3    // 11
2,4,3,1    // 12
3,1,2,4    // 13
3,1,4,2    // 14
3,2,1,4    // 15
3,2,4,1    // 16
3,4,1,2    // 17
3,4,2,1    // 18
4,1,2,3    // 19
4,1,3,2    // 20
4,2,1,3    // 21
4,2,3,1    // 22
4,3,1,2    // 23
4,3,2,1   // 24

Now let's retrieve the permutations for the indexes:

(instead of calling it a permutation I will now call it an "attack order"):

Index1: 7  Attack order: 2,1,3,4
Index2: 22  Attack order: 4,2,3,1
Index3: 12  Attack order: 1,3,2,4
Index4: 10  Attack order: 2,3,4,1
Index5: 4  Attack order: 1,3,4,2
Index6: 9  Attack order: 2,3,1,4
Index7: 11  Attack order: 2,4,1,3
Index8: 5  Attack order: 2,1,4,3

Now the computer program can perform the combat.

However perform I explain further how a combat would be performed I will try 
to answer the rest of your questions:

", what you consider a number,"

A number is a tag uppon an object. The object could be a ship, a tank, an 
airplane, or whatever the real world scenerio is behind the problem.

I was considering mentioning this, but I thought it let it out just for the 
fun of it. Consider "number" the number of the object performing the attack.

Perhaps giving some context to these numbers might have helped to imagine 
the problem better. If this is indeed the case then perhaps a little "sorry" 
might be in place.

Numbers usually don't fight each other... so that concept/idea is kinda 
weird/vague... I was kinda hoping that people would find it a bit funny... 
but could see through that and understand by themselfes that these
numbers probably could represent a solder,tank,plane,ship and so forth. I 
also want to conceal what this program will be used for just to thart any 
possible/potential enemy that might read these postings ! :)
But if you must know it's written to help me understand which attack 
strategies work better and which work less well in a specific online game... 
but that's all I will say about it... perhaps you can now already interfere 
which game it might be ;)

"nor how your result table works"

Hmmm fair enough point I guess perhaps that wasn't so clearly described. If 
you want me to give you an exact definition then fair enough.

The results table is simply the number of numbers on each team dimensionally 
multiplied by the number of permutations (=attack orders).

So since momentarily the team consists out of 4 contenders/numbers and since 
attack orders are 24 the entire result table would be a 2D table: 4 by 24.

"you don't identify x vs y"

X is a group
Y is a group

I think this was pretty clearly defined ? However I can see how your 
question might be related to the victory table since you seem to be unsure 
what the rows and columns represent.

I was kinda hoping that you would automatically understand what it ment 
based on the description of "numbers fighting each others". But perhaps that 
description was too vague and too multi-interpretable. Or simply too 
confusing/unclear to what it relates.

To put it simply: The victory table describes the "number to number" attack 
chance. Which means the individual chance of an individual number attacking 
another individual number. In other words the victory table has nothing to 
do with the groups themselfes.

Since there are only two groups I don't see how a victory table could be 
related to the groups themselfes. However the groups were tag as X/Y also to 
prevent confusing with the speaking term "a table", "a bank", "a bike".

However now that I see me writing this table if you plot the table versus 
the x-axis and the y-axis... and put some numbers on it... it might have 
been more clear. I am afraid that apperently it's still not clear ?!

So I will have to explain harder... at least for you... maybe for others 
two... perhaps an example will help to clearify how the Victory Table is to 
be used.

Also perhaps I should have mentioned that each number is also a type. That 
might have made it more clear how the envision the victory table.

So to help you understand I will now describe that number 1 could represent 
a horse, number 2 could represent a tank, number 3 could represent a plane, 
number 4 could represent a ship.

What the numbers/types truely represent will remain concealed for now to 
thart any enemies reading this text :)

However I hope by "applieing" types/classes to the numbers that maybe now it 
is starting to make some sense to you.

The victory table basically describes what the chance is that for example a 
horse will win from a tank.

Or what the chance will be of a tank winning against a ship.

Or what the chance will be of a plane winning against a ship.

So basically the Y-Axis describes the source=attacker

So basically the X-Axis describes the destination=target

Finally the chance in the table itself thus describes the chance of the 
attacker killing the target.

(The chance could also be interpreted as "damage done" which is actually 
what my code uses for now).

", and since it is not diagonally symmetric that makes a difference... "

I don't quite understand this sentence of yours... a bit weird... but I 
guess this was because of your confusion and because you trying to sum the 
entries in the victory table for some reason, which is because of 
misunderstand... it's not necessary, I hope to have cleared that up above 
and will continue with an example below.

"nor do the rows/columns sum to 100%)"

Not required.

Finally an example.

Let's say a horse attacks a tank.

What would it's chance of winning be ?

Let's first define the numbering as describes above:
Horse=1
Tank=2
Plane=3
Ship=4

Let's re-examine our victory table:

Let's apply some row/column numbering perhaps that will clearify it a bit 
for you.

    1   2   3   4
1: 50,  3, 80, 70
2: 90, 60, 20, 40
3: 30, 90, 55, 65
4: 75, 90, 98, 60

Now the program wants to know what is the chance of a horse defeating a tank 
?

So horse = 1

So proceed to row.

So tank = 3

So proceed to column 3.

This gives the horse a winning chance of 80%.

Ofcourse this makes completely nosense with these horse, tank, plane,ship 
descriptions.

But I hope you get the idea.


So finally what could a possible solution program do to calculate outcome of 
battle ?!? Let's proceed with the example:

Index1: 7  Attack order: 2,1,3,4
Index2: 22  Attack order: 4,2,3,1
Index3: 12  Attack order: 1,3,2,4
Index4: 10  Attack order: 2,3,4,1
Index5: 4  Attack order: 1,3,4,2
Index6: 9  Attack order: 2,3,1,4
Index7: 11  Attack order: 2,4,1,3
Index8: 5  Attack order: 2,1,4,3


Index 1,2,3,4 belongs to group X
index 5,6,7,8 belongs to group Y (5,6,7,8 renumbered as 1,2,3,4)


>From the information above it's now clear how group X will attack group Y 
and how group Y will attack group X.

X1 will first attack Y2
X2 will first attack Y4
X3 will first attack Y1
X4 will first attack Y2

Y1 will first attack X1
Y2 will first attack X2
Y3 will first attack X2
Y4 will first attack X2

>From looking at this first round of attacks from above it's already pretty 
clear that Group Y has "chosen" to attack X2 with 3 objects. Thus it's very 
likely that X2 will be destroyed.

Funny enough groupX is also attacking Y2 but only with 2 objects. So Y2 is 
also likely to be destroyed but a bit less likely.

How the program now proceeds with calculations is open to debate.

There are two possibilities:

Either the program considers X2 destroyed if the chances are subtracted from 
100 and it goes below 0. (The chances are lookup in the victory table as 
describes in the example above for horse vs tank).

This I would call the "dynamic" approach... it complexifies the problem 
somewhat... a number which is destroyed cannot further attack so it will not 
take part in the next round of attacks.

However another approach could be taken which stays more true to the concept 
of a "decision tree". Instead of subtracting the chances, the chances could 
also be multiplied... this will reduce the chance of X2 being alive, but it 
is still allowed to participate in the fight assuming
it had a very low chance of surviving but might have miracously survived. To 
compensate for this perhaps further attacks from X2 should have it's chance 
of success reduced... to make the computation a bit more realistic but still 
computationally easy to do: just a multiplication, no branches.

So for now I will explain how my program proceeds. Without actually looking 
up the chances... which would be a bit much work to do... I will simply 
assume that X2 died and perhaps Y2 too.

So the next round commences as follows:

(attack targets/indexes acquired from attack order from above):

X1 will first attack Y1
X2 (destroyed)
X3 will first attack Y3
X4 will first attack Y3

Y1 will first attack X1
Y2 (destroyed)
Y3 will first attack X4
Y4 will first attack X1

Perhaps after this second round Y3 is destroyed, perhaps X1 is destroyed

So round 3 commences:

X1 (destroyed)
X2 (destroyed)
X3 will first attack Y2
X4 will first attack Y4

Y1 will first attack X4
Y2 (destroyed)
Y3 (destroyed)
Y4 will first attack X4

Apperently group Y had the more lucky permutation of attack orders, here 
they jointly attack 4 so it's likely 4 was described which gives them a 
better chance of winning since now they are +1 vs the enemy, while Y2/Y4 are 
only damaged but not yet destroyed.

Let's see what happens in potentially the final round:

X1 (destroyed)
X2 (destroyed)
X3 will first attack Y4
X4 (destroyed)

Y1 will first attack X2, cycles to X1, cycles to X3
Y2 (destroyed)
Y3 (destroyed)
Y4 will first attack X3

Here my program if I remember correctly will detect that enemy X2 is already 
destroyed so it's starts cycling/wrapping it's attack order index to find 
which one it should attack next until it finds X3 still alive and attacks 
it.
(First Y1 tried X1 according to it's attack order "table" but X1 is also 
already destroyed so it proceeds to X3).

>From this "permutation" example it seems likely that X3 was destroyed and 
group Y might be the winner.

However if group Y is truely the winner will depend on the actually victory 
chances... perhaps something weird occured and one of the objects was easily 
destroyed by a high victory chance... even though it was 1 vs 1... thus this 
might have completely altered the outcome of battle.

So that's what the computer program will have to do/compute.

Also finally let's look at the results table.

For this outcome the results table would be adjusted as follows:

Group Y won, so no adjustment. Only Group X is examined, since it's a mirror 
of each other.

However let's assume that the exact same battle happens but vice versa. 
Group X/Y swapped.

In that case the results table would be adjusted as follows:

Index1: 7   Victories +1
Index2: 22  Victories +1
Index3: 12  Victories +1
Index4: 10  Victories +1

This basically means that the permutation 7 for object 1 has +1 victories 
independently record of what the rest did.
This basically means that the permutation 22 for object 2 has +1 victories 
independently record of what the rest did.
This basically means that the permutation 12 for object 3 has +1 victories 
independently record of what the rest did.
This basically means that the permutation 10 for object 4 has +1 victories 
independently record of what the rest did.

So basically in C/Delphi like terminology:
ObjectPermutationVictories[1,7]++
ObjectPermutationVictories[2,22]++
ObjectPermutationVictories[3,12]++
ObjectPermutationVictories[4,10]++


*** Continueing with the rest, unrelated to above ***

(By the way I made a little typo in this weird text below 3th should have 
been 5th, I thought 3 bits would be enough but it actually needed 5 bits):

"
Just the 8 index loops already cost a lot of instructions. Since there are
only 24 permutation it would be enough to store it in 5 bits. Perhaps a
rounded floating point which increases by 1/24 might be enough to trigger
the 5th bit from incrementing when it actually needs to.
2x2x2x2x2 = 32  (it should increment bit 6 when the 5 bits reach 24).
So the idea here was to create 8 indexes from just 1 index being incremented
to create the 8 combinations of indexes "instruction cheaply".
Not sure if this will work, just an idea I might try :)
"

If you don't understand this crazy idea above in quotations then you can 
safely ignore it. It is not used it was an experimental thought how things 
might be done differently in regards to advancing the indexes.

***


Now continueing with the rest of your questions


"
>>> import itertools
>>> for x in itertools.permutations("1234"):
... xout = "".join(x)
... for y in itertools.permutations("1234"):
... print xout, "".join(y)
...
1234 1234
1234 1243
1234 1324
1234 1342
1234 1423
1234 1432
1234 2134
1234 2143
1234 2314
1234 2341
"

I don't quite understand that python code, but I can imagine what it might 
be doing... but it's probably not correct, it's too simplistic. Or perhaps 
this was just to generate the permutations.

As I wrote above... generating the permutations is not really the problem, 
they can be looked up from the permutation table.

"
Replace the print with something to compute just the win/loss stuff for the
single x vs y, and save the result off -- since I couldn't work out your
scoring I can't say if a dictionary keyed by the permutation makes sense...
"

For now I will not reply to this...  there seems to be some understanding in 
what you wrote... but for now I will wait to see if this new reply of mine 
will have clearified a lot for you and thus perhaps I do not need to respond 
to the text above.


"
And are you generating random numbers to determine the winner/loser?
"

No, brute force is applied to calculate which group wins, in case you ment 
which group wins/losses. If you ment which number wins, then that is giving 
by a chance lookup and perhaps a health variable per object which is 
decremented. However you are free to decide what
to do... kill of numbers ? keep them alive ? Use a different way of 
reasoning what should happen to them. The victory chance should be applied 
somehow though. It definetly should influence the outcome of battle somehow.

" How many cycles -- loop on random values until one or the other is "dead". 
"

A "while loop" which examines the "alive status" of each number/object is 
used to examine if the battle came to an end. As long as each team has an 
alive member the fight/combat continues.
Once the numbers of one group are completely destroyed, the other group 
wins. In case both groups destroy each other at the same time, it's a draw.

"What is the statistics? How many combats the digits took (ie, number of 
cycles of random)... How many it took for all four digits to die?"

Nice question.

This depends on the approach taken. I decided to take a "natural" feeling 
approach, which I call the "dynamic approach". The combat for each 
possibility is continued until one group dies. Therefore it's hard to say 
what the exact number of "rounds" or "cycles" as you call them it will take.

However it will probably be something like 4 or 5 cycles. This is kind of an 
interesting question. Some additional statistical analysis variables could 
be added to the code to track what the average number of rounds is, what the 
minimum was and what the maximum was !

Thanks for this feedback it's kinda interesting ! ;)

Bye,
  Skybuck. 


From __peter__ at web.de  Thu Dec 15 07:16:33 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 15 Dec 2016 13:16:33 +0100
Subject: [OT] "Invisible posts",
 was: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
Message-ID: <o2u1iu$und$1@blaine.gmane.org>

skybuck2000 at hotmail.com wrote:

> I received a reply from somebody on my ISP newsserver. Apperently his
> reply is not visible on google groups. I wonder why, maybe it's a banned
> troll or something, but perhaps not.

No, that's Dennis Lee Bieber who doesn't want his posts to be kept, and 
seems to be the only one to post here with the X-No-Archive flag set.



From rhodri at kynesim.co.uk  Thu Dec 15 08:51:59 2016
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 15 Dec 2016 13:51:59 +0000
Subject: Python3, column names from array - numpy or pandas
In-Reply-To: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>
References: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>
Message-ID: <8567441e-3cf5-f680-e402-c239d13710fe@kynesim.co.uk>

On 15/12/16 01:56, renjith madhavan wrote:
> I have a dataset in the below format.
>
> id	A	B	C	D	E
> 100	1	0	0	0	0
> 101	0	1	1	0	0
> 102	1	0	0	0	0
> 103	0	0	0	1	1
>
> I would like to convert this into below:
> 100, A
> 101, B C
> 102, A
> 103, D E
>
> How do I do this ? I tried numpy argsort but I am new to Python and finding this challenging.
> Appreciate any help in this.
>

Numpy or pandas?  Neither, this is a straightforward bit of text 
manipulation you can do without needing to import anything.  I wouldn't 
bother considering either unless your dataset is massive and speed is 
anything of an issue.

with open("data.txt") as datafile:
     # First line needs handling separately
     line = next(datafile)
     columns = line.split()[1:]
     # Now iterate through the rest
     for line in datafile:
         results = []
         for col, val in zip(columns, line.split()[1:]:
              if val == "1":
                  results.append(col)
         print("{0}, {1}".format(data[0], " ".join(results)))

Obviously there's no defensive coding for blank lines or unexpected data 
in there, and if want to use the results later on you probably want to 
stash them in a dictionary, but that will do the job.

-- 
Rhodri James *-* Kynesim Ltd


From renjithmadhavan at gmail.com  Thu Dec 15 08:56:41 2016
From: renjithmadhavan at gmail.com (renjith madhavan)
Date: Thu, 15 Dec 2016 05:56:41 -0800 (PST)
Subject: Python3, column names from array - numpy or pandas
In-Reply-To: <0418ef59-1c34-4085-8fcd-469b1dd7ed13@googlegroups.com>
References: <60566d14-ed64-41ce-b654-15ee683973ab@googlegroups.com>
 <0418ef59-1c34-4085-8fcd-469b1dd7ed13@googlegroups.com>
Message-ID: <6e8712da-99af-419a-911f-10349908505e@googlegroups.com>

Thank you for the reply. 
I tried that, I am trying to do this.

The context is I am trying to find mapk ( k = 3 ) for this list.
A, B , C, D and E are product names.

If I am trying manually I will do something like this.

TRUTH = [[A], [B,C], [A], [D,E]]
and if my prediction is :
PRED=[[B,A, D], [A,C,B], [A,B,C], [B,D,E]]

map3(truth, pred, 3)

 
How do I convert my input truth values in the TRUTH format.
Should I be looking for "boolean indexing" for this case.
id        A        B        C        D        E 
    100        1        0        0        0        0 
    101        0        1        1        0        0 
    102        1        0        0        0        0 
    103        0        0        0        1        1 
    ''') 


From ohiomcbd at gmail.com  Thu Dec 15 11:11:51 2016
From: ohiomcbd at gmail.com (Jed Mack)
Date: Thu, 15 Dec 2016 11:11:51 -0500
Subject: Problem running Python 3.5.2 on school network PC
Message-ID: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>

We are having a problem running Python 3.5.2 on Windows 10 x64 computers,
which are members of a school network.



The program seems to install correctly, but when we try to run the program
it stops and give an error message saying:



*Fatal Python error: Py_Initialize: unable to load the file system codec*

*Traceback (most recent call last):*

*  File "C:\Program Files\Python35\lib\encodings\__init__.py", line 31, in<
module>*

*zipimport.ZipImportError: can't find module 'codecs'*





On a Windows 7 PCs, on the same network, the program runs with no problems.



We have no one here who is familiar with Python.  Do you have any
additional information on this error, and suggestions for fixing it?



We have a teacher who needs this program on Windows 10 PCs for students to
use beginning January 3.



Thanks,

Jed Mack


From random832 at fastmail.com  Thu Dec 15 11:49:06 2016
From: random832 at fastmail.com (Random832)
Date: Thu, 15 Dec 2016 11:49:06 -0500
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <sp355cl79sr0oaeqtrkoonusir9kt88r08@4ax.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <sp355cl79sr0oaeqtrkoonusir9kt88r08@4ax.com>
Message-ID: <1481820546.1708171.820205065.40FB1422@webmail.messagingengine.com>

On Thu, Dec 15, 2016, at 08:31, Dennis Lee Bieber wrote:
> 	As for my posts disappearing: I run with "X-NoArchive" set. I have from
> before Google absorbed DejaNews. Back then, most news-servers expired
> posts
> on some periodic basis (my ISP tended to hold text groups for 30 days or
> so, binaries groups cycled in less than 36 hours). When DejaNews arrived,
> many objected to the non-expiration facet:
> https://en.wikipedia.org/wiki/Google_Groups#Deja_News

As I recall, what most people *actually* said they objected to was the
fact that they profited from showing ads. X-No-Archive was just a
convenient blunt weapon to hit them with.


From steve+python at pearwood.info  Thu Dec 15 11:53:58 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 16 Dec 2016 03:53:58 +1100
Subject: Unicode script
Message-ID: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>

Suppose I have a Unicode character, and I want to determine the script or
scripts it belongs to.

For example:

U+0033 DIGIT THREE "3" belongs to the script "COMMON";
U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".


Is this information available from Python?


More about Unicode scripts:

http://www.unicode.org/reports/tr24/
http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Thu Dec 15 12:06:25 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 16 Dec 2016 04:06:25 +1100
Subject: Mapping with continguous ranges of keys
Message-ID: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>

I have some key:value data where the keys often are found in contiguous
ranges with identical values. For example:

{1: "foo",
 2: "foo",
 3: "foo",
 # same for keys 4 through 99
 100: "foo",
 101: "bar",
 102: "bar",
 103: "foobar",
 104: "bar",
 105: "foo", 
}


So in this case, the keys 1 through 100, plus 105, all have the same value.

I have about a million or two keys, with a few hundred or perhaps a few
thousand distinct values. The size of each contiguous group of keys with
the same value can vary from 1 to perhaps a hundred or so.

Has anyone dealt with data like this and could give a recommendation of the
right data structure to use?

The obvious way is to explicitly list each key, in a regular dict. But I
wonder whether there are alternatives which may be better (faster, more
memory efficient)?


Thanks,


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From darcy at vex.net  Thu Dec 15 12:24:51 2016
From: darcy at vex.net (D'Arcy Cain)
Date: Thu, 15 Dec 2016 12:24:51 -0500
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <6b403f3e-9863-1656-95c8-9fbf406603f3@vex.net>

On 2016-12-15 12:06 PM, Steve D'Aprano wrote:
> I have about a million or two keys, with a few hundred or perhaps a few
> thousand distinct values. The size of each contiguous group of keys with
> the same value can vary from 1 to perhaps a hundred or so.

There isn't enough info in your post to be sure but if those values are 
constant then you might be able to subclass dict and write a new 
__getitem__ that checks for specific ranges and calls the superclass 
only if not in the known ranges.  For example:

class MyDict(dict):
   def __getitem__(self, key):
     if isinstance(key, int) and key >= 1 and key <= 100:
       return "foo"
     return dict.__getitem__(self, key)

Obviously that middle section can be as complex as you need.

-- 
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:darcy at Vex.Net
VoIP: sip:darcy at Vex.Net


From tomuxiong at gmx.com  Thu Dec 15 12:27:45 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Thu, 15 Dec 2016 09:27:45 -0800
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>

On 12/15/2016 09:06 AM, Steve D'Aprano wrote:
> Has anyone dealt with data like this and could give a recommendation of the
> right data structure to use?
>

I haven't dealt with a data structure exactly like this, but it's 
basically a sparse array. (I'm sure it has a name somewhere in the 
academic literature, but I couldn't find it with a quick search right 
now...)

My solution to what you're asking for would be to have a list of 
key-value pairs, only adding a key to the list if it "changes" the 
value. I.e. your data structure would be this:

l = [
     (1, "foo"),
     (101, "bar"),
     (103, "foobar"),
     (104, "bar"),
     (105, "foo"),
]

Then the only thing you need to do is define the lookup function. I 
would basically just do a binary search on the first values in the 
tuples. I.e. if "n" is your integer, you check if the middle values of 
the list l has it's first element as less than or greater than your 
value. Then you split l in two and do the same thing. Do this until you 
either find your value, or you find a value less than your value with 
the added property that the next value is greater than your value. After 
that you spit out the final second value.

There might be better ways to find the keys, but I think this approach 
is probably your best bet.

Cheers,
Thomas


From __peter__ at web.de  Thu Dec 15 12:54:48 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 15 Dec 2016 18:54:48 +0100
Subject: Mapping with continguous ranges of keys
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2uld7$pk1$1@blaine.gmane.org>

Steve D'Aprano wrote:

> I have some key:value data where the keys often are found in contiguous
> ranges with identical values. For example:
> 
> {1: "foo",
>  2: "foo",
>  3: "foo",
>  # same for keys 4 through 99
>  100: "foo",
>  101: "bar",
>  102: "bar",
>  103: "foobar",
>  104: "bar",
>  105: "foo",
> }
> 
> 
> So in this case, the keys 1 through 100, plus 105, all have the same
> value.
> 
> I have about a million or two keys, with a few hundred or perhaps a few
> thousand distinct values. The size of each contiguous group of keys with
> the same value can vary from 1 to perhaps a hundred or so.
> 
> Has anyone dealt with data like this and could give a recommendation of
> the right data structure to use?
> 
> The obvious way is to explicitly list each key, in a regular dict. But I
> wonder whether there are alternatives which may be better (faster, more
> memory efficient)?

Use a list, either directly if there are no big holes

>>> r = range(10**5)
>>> sys.getsizeof(list(r))/sys.getsizeof(dict(zip(r, r)))
0.14306676635590074


or indirectly:

ranges_list = [
    1,
    101,
    103,
    104,
    105,
    106,
]

index_to_value = {
    1: "foo",
    2: "bar",
    3: "foobar",
    4: "bar",
    5: "foo",
}

def get(key, default="missing"):
    index = bisect.bisect(ranges_list, key)
    return index_to_value.get(index, default)




From eryksun at gmail.com  Thu Dec 15 13:01:58 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 15 Dec 2016 18:01:58 +0000
Subject: Unicode script
In-Reply-To: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CACL+1asC7K26YJ4wK8ctkCdKeq+YMP25FUiMD6Ur73uG0-+Vng@mail.gmail.com>

On Thu, Dec 15, 2016 at 4:53 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> Suppose I have a Unicode character, and I want to determine the script or
> scripts it belongs to.
>
> For example:
>
> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>
> Is this information available from Python?

Tools/makunicodedata.py doesn't include data from "Scripts.txt". If
adding an external dependency is ok, then you can use PyICU. For
example:

    >>> icu.Script.getScript('\u0033').getName()
    'Common'
    >>> icu.Script.getScript('\u0061').getName()
    'Latin'
    >>> icu.Script.getScript('\u03be').getName()
    'Greek'

There isn't documentation specific to Python, so you'll have to figure
things out experimentally with reference to the C API.

http://icu-project.org/apiref/icu4c
http://icu-project.org/apiref/icu4c/uscript_8h.html


From joel.goldstick at gmail.com  Thu Dec 15 13:03:39 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Thu, 15 Dec 2016 13:03:39 -0500
Subject: Unicode script
In-Reply-To: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPM-O+x8f+hNx7UbTg+OA8AfN9BEfRFqPGrPewnW9D9d6n_-7A@mail.gmail.com>

I think this might be what you want:
https://docs.python.org/3/howto/unicode.html#unicode-properties

On Thu, Dec 15, 2016 at 11:53 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> Suppose I have a Unicode character, and I want to determine the script or
> scripts it belongs to.
>
> For example:
>
> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>
>
> Is this information available from Python?
>
>
> More about Unicode scripts:
>
> http://www.unicode.org/reports/tr24/
> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt
>
>
> --
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.
>
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From pkpearson at nowhere.invalid  Thu Dec 15 13:05:13 2016
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 15 Dec 2016 18:05:13 GMT
Subject: OT - "Soft" ESC key on the new MacBook Pro
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
Message-ID: <ebg4apF76f9U1@mid.individual.net>

On Wed, 14 Dec 2016 11:50:30 -0600, Skip Montanaro wrote:
> On Wed, Dec 14, 2016 at 11:40 AM, Peter Pearson
><pkpearson at nowhere.invalid> wrote:
>> Train your fingers to use C-[.
>
> As I recall, the location of the Ctrl key was one of the differences
> between Sun and PC101 keyboards. Doesn't matter so much now, as Sun
> has gone the way of the dodo, but it moved around more for me than ESC
> over the years.

Absolutely right.  Random migrations of the Ctrl key annoyed so many
of us set-in-our-ways geezers that Linux distributions always seem to
come with an easily activated option to put the Ctrl key where it
belongs, namely to the left of the A, right where God put it on Adam's
ASR 33.

-- 
To email me, substitute nowhere->runbox, invalid->com.


From tjreedy at udel.edu  Thu Dec 15 13:05:31 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 13:05:31 -0500
Subject: =?UTF-8?Q?Re:_Python_constructors_have_particular_semantics=2c_and_?=
 =?UTF-8?B?4oCYRm9vLl9faW5pdF9f4oCZIGRvZXNuJ3QgcXVhbGlmeQ==?=
In-Reply-To: <2311421.k3LOHGUjKi@PointedEars.de>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
 <mailman.38.1481587045.2322.python-list@python.org>
 <2311421.k3LOHGUjKi@PointedEars.de>
Message-ID: <o2um19$l8m$1@blaine.gmane.org>

On 12/14/2016 11:14 PM, Thomas 'PointedEars' Lahn wrote:

> According to <https://docs.python.org/3/tutorial/classes.html#method-objects>, ?Foo.__init__? is _not_ an instance method.  Were it an instance
> method, the following would not happen:

This link points to subsection 9.3.4. Method Objects

> | >>> class Foo:
> | ...     def __init__ (self):
> | ...         pass
> | ...
> | >>> Foo.__init__.__self__
> | Traceback (most recent call last):
> |   File "<stdin>", line 1, in <module>
> | AttributeError: 'function' object has no attribute '__self__'

You have misread the docs.  Foo.__init__ is the function. 
Foo().__init__ is a method object with the attribute __self__.  You 
omitted the ()s.

 >>> class C:
	def __init__(self): pass

	
 >>> ci = C().__init__
 >>> ci.__self__
<__main__.C object at 0x000001E38A750E80>

> Because ?Instance method objects have attributes, too: m.__self__ is the
> instance object with the method m() [?]?.

This line is from section 9.7. Odds and Ends. In this quote, 'm' is a 
method object, the result of 'instance.method, not a function.  In 
Python 2, 'm' would have been called a 'bound method', as opposed to an 
unbound method.  Since the latter were eliminated in 3.x, the adjective 
is no longer needed.



-- 
Terry Jan Reedy




From python at mrabarnett.plus.com  Thu Dec 15 13:06:49 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 15 Dec 2016 18:06:49 +0000
Subject: Unicode script
In-Reply-To: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <392bb301-0e54-3607-7b7f-3c13e7bb6b9b@mrabarnett.plus.com>

On 2016-12-15 16:53, Steve D'Aprano wrote:
> Suppose I have a Unicode character, and I want to determine the script or
> scripts it belongs to.
>
> For example:
>
> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>
>
> Is this information available from Python?
>
>
> More about Unicode scripts:
>
> http://www.unicode.org/reports/tr24/
> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt
>
>
Interestingly, there's issue 6331 "Add unicode script info to the 
unicode database". Looks like it didn't make it into Python 3.6.



From tjreedy at udel.edu  Thu Dec 15 13:28:46 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 13:28:46 -0500
Subject: Problem running Python 3.5.2 on school network PC
In-Reply-To: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
References: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
Message-ID: <o2uncs$r9q$1@blaine.gmane.org>

On 12/15/2016 11:11 AM, Jed Mack wrote:
> We are having a problem running Python 3.5.2 on Windows 10 x64 computers,
> which are members of a school network.
>
> The program seems to install correctly,

Is Python installed on each machine or on a network server?
Rather precisely, how was Python installed?
What installation package was downloaded?  From where?
How was it run?

 > but when we try to run the program
> it stops and give an error message saying:
>
> *Fatal Python error: Py_Initialize: unable to load the file system codec*
>
> *Traceback (most recent call last):*
>
> *  File "C:\Program Files\Python35\lib\encodings\__init__.py", line 31, in<
> module>*

The line is
     import codecs

> *zipimport.ZipImportError: can't find module 'codecs'*

This says that the stdlib is in the form of a zipfile instead of the 
normal /lib directory.  I don't know how one gets that result on 
Windows.  It appears that the zipped library is missing the file.

> On a Windows 7 PCs, on the same network, the program runs with no problems.

Same Python version?  Same installation file?  Run the same way?

> We have no one here who is familiar with Python.  Do you have any
> additional information on this error, and suggestions for fixing it?

> We have a teacher who needs this program on Windows 10 PCs for students to
> use beginning January 3.

We appreciate that you are trying to use current Python to teach.  I 
hope we are able to solve the problem, given more information.

-- 
Terry Jan Reedy



From best_lay at yahoo.com  Thu Dec 15 13:31:11 2016
From: best_lay at yahoo.com (Wildman)
Date: Thu, 15 Dec 2016 12:31:11 -0600
Subject: Problem running Python 3.5.2 on school network PC
References: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
 <mailman.92.1481821534.2322.python-list@python.org>
Message-ID: <ea-dnXEi4PXyfM_FnZ2dnUU7-KudnZ2d@giganews.com>

On Thu, 15 Dec 2016 11:11:51 -0500, Jed Mack wrote:

> We are having a problem running Python 3.5.2 on Windows 10 x64 computers,
> which are members of a school network.
> 
> 
> 
> The program seems to install correctly, but when we try to run the program
> it stops and give an error message saying:
> 
> 
> 
> *Fatal Python error: Py_Initialize: unable to load the file system codec*
> 
> *Traceback (most recent call last):*
> 
> *  File "C:\Program Files\Python35\lib\encodings\__init__.py", line 31, in<
> module>*
> 
> *zipimport.ZipImportError: can't find module 'codecs'*
> 
> 
> 
> 
> 
> On a Windows 7 PCs, on the same network, the program runs with no problems.
> 
> 
> 
> We have no one here who is familiar with Python.  Do you have any
> additional information on this error, and suggestions for fixing it?
> 
> 
> 
> We have a teacher who needs this program on Windows 10 PCs for students to
> use beginning January 3.
> 
> 
> 
> Thanks,
> 
> Jed Mack

You might try adding the installation folders to the path, if it
hasn't been done already.  Change 'python' to the actual folder name.

C:\python
C:\python\Lib\site-packages
C:\python\Scripts

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From mail at timgolden.me.uk  Thu Dec 15 13:34:21 2016
From: mail at timgolden.me.uk (Tim Golden)
Date: Thu, 15 Dec 2016 18:34:21 +0000
Subject: Problem running Python 3.5.2 on school network PC
In-Reply-To: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
References: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
Message-ID: <33d71e3c-903f-ec27-7f25-a462a46abd73@timgolden.me.uk>

On 15/12/2016 16:11, Jed Mack wrote:
> We are having a problem running Python 3.5.2 on Windows 10 x64 computers,
> which are members of a school network.
>
> The program seems to install correctly, but when we try to run the program
> it stops and give an error message saying:
>
> *Fatal Python error: Py_Initialize: unable to load the file system codec*
>
> *Traceback (most recent call last):*
>
> *  File "C:\Program Files\Python35\lib\encodings\__init__.py", line 31, in<
> module>*
>
> *zipimport.ZipImportError: can't find module 'codecs'*
>
> On a Windows 7 PCs, on the same network, the program runs with no problems.
>
> We have no one here who is familiar with Python.  Do you have any
> additional information on this error, and suggestions for fixing it?
>
> We have a teacher who needs this program on Windows 10 PCs for students to
> use beginning January 3.

This most often seems to happen when there is another instance of Python 
installed (or, possibly, previously installed and partly uninstalled). 
Especially: look for environment variables which could be left over from 
such an installation, particularly PYTHONHOME.

NB it's possible for Python to have been installed as part of some OEM 
software, ie not as part of your school's own installation protocol.

TJG


From tjreedy at udel.edu  Thu Dec 15 13:34:32 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 13:34:32 -0500
Subject: Reading python list as a newsgroup (was ...)
In-Reply-To: <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
Message-ID: <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>

On 12/15/2016 6:23 AM, skybuck2000 at hotmail.com wrote:

> Anyway... my ISP has problems accessing their newsserver. ...

If you want to read python-list as a news group, you might try 
news.gmane.org.  About once a year, it goes down for a few hours to a 
day, but has otherwise been dependable.  I found it when my ISP dropped 
newsgroup access around a decade ago.



From ian.g.kelly at gmail.com  Thu Dec 15 14:04:31 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Thu, 15 Dec 2016 12:04:31 -0700
Subject: =?UTF-8?Q?Re=3A_Python_constructors_have_particular_semantics=2C_a?=
 =?UTF-8?Q?nd_=E2=80=98Foo=2E=5F=5Finit=5F=5F=E2=80=99_doesn=27t_qualify?=
In-Reply-To: <o2um19$l8m$1@blaine.gmane.org>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
 <mailman.38.1481587045.2322.python-list@python.org>
 <2311421.k3LOHGUjKi@PointedEars.de> <o2um19$l8m$1@blaine.gmane.org>
Message-ID: <CALwzidkQ_HryC_2zieEYM43usHHO8ReomaCMSUCVqBNVNyRktA@mail.gmail.com>

On Thu, Dec 15, 2016 at 11:05 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 12/14/2016 11:14 PM, Thomas 'PointedEars' Lahn wrote:
>
>> According to
>> <https://docs.python.org/3/tutorial/classes.html#method-objects>,
>> ?Foo.__init__? is _not_ an instance method.  Were it an instance
>> method, the following would not happen:
>
>
> This link points to subsection 9.3.4. Method Objects
>
>> | >>> class Foo:
>> | ...     def __init__ (self):
>> | ...         pass
>> | ...
>> | >>> Foo.__init__.__self__
>> | Traceback (most recent call last):
>> |   File "<stdin>", line 1, in <module>
>> | AttributeError: 'function' object has no attribute '__self__'
>
>
> You have misread the docs.  Foo.__init__ is the function. Foo().__init__ is
> a method object with the attribute __self__.  You omitted the ()s.

I think this is actually the point that Thomas was making. He was
responding to the assertion that "Foo.__init__" is an instance method
and demonstrating that it's false because Foo is the class, not an
instance of Foo.

If I've observed anything about Thomas over the years he's been
posting here, it's that he's extremely literal and will pick apart
tiny irrelevant nuances to the point of being (or at least appearing)
trollish.


From tjreedy at udel.edu  Thu Dec 15 14:45:15 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 14:45:15 -0500
Subject: Unicode script
In-Reply-To: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2urs9$5bj$1@blaine.gmane.org>

On 12/15/2016 11:53 AM, Steve D'Aprano wrote:
> Suppose I have a Unicode character, and I want to determine the script or
> scripts it belongs to.
>
> For example:
>
> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>
> Is this information available from Python?



Yes, though not as nicely as you probably want.  (Have you searched for 
existing 3rd party modules?)  As near as I can tell, there is no direct 
'script' property in the unicodedatabase.

Option 1: unicodedata module, from char name

 >>> import unicodedata as ucd
 >>> ucd.name('\u03be')
'GREEK SMALL LETTER XI'
 >>> ucd.name('\u0061')
'LATIN SMALL LETTER A'

In most cases, the non-common char names start with a script name.
In some cases, the script name is 2 or w words.

 >>> ucd.name('\U00010A60')
'OLD SOUTH ARABIAN LETTER HE'

In a few cases, the script name is embedded in the name.
 >>> ucd.name('\U0001F200')
'SQUARE HIRAGANA HOKA'

Occasionally the script name is omitted.
 >>> ucd.name('\u3300')
'SQUARE APAATO'  # Katakana

To bad the Unicode Consortium did not use a consistent name scheme:
script [, subscript]: character

LATIN: SMALL LETTER A
ARABIAN, OLD SOUTH: LETTER HE
KATAKANA: SQUARE APAATO

> More about Unicode scripts:
>
> http://www.unicode.org/reports/tr24/
> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt

Option 2: Fetch the above Scripts.txt.

Suboption 1: Turn Scripts.txt into a list of lines.  The lines could be 
condensed to codepoint or codepoint range, script.  Write a function 
that takes a character or codepoint and linearly scans the list for a 
matching line.  This makes each lookup O(number-of-lines).

Suboption 2. Turn Scripts.txt into a list of scripts, with codepoint 
being the index.  This takes more preparation, but makes each lookup 
O(1).  Once the preparation is done, the list could be turned into a 
tuple and saved as a .py file, with the tuple being a compiled constant 
in a .pyc file.

To avoid bloat, make sure that multiple entries for a script use the 
same string object instead of multiple equal strings.  (CPython string 
interning might do this automatically, but cross-implementation code 
should not depend on this.)  The difference is

scripts = [..., 'Han', 'Han', 'Han', ...] # multiple strings
versus
HAN = 'Han'
scripts = [..., HAN, HAN, HAN, ...]  # multiple references to one string

On a 64 bit OS, the latter would use 8 x defined codepoints (about 
200,000) bytes.  Assuming such does not already exits, it might be worth 
making such a module available on PyPI.

> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt

Essentially, ditto, except that I would use a dict rather than a 
sequence as there are only about 400 codepoints involved.

-- 
Terry Jan Reedy




From rbrowning1 at mmm.com  Thu Dec 15 15:24:28 2016
From: rbrowning1 at mmm.com (Rhesa Browning)
Date: Thu, 15 Dec 2016 20:24:28 +0000
Subject: Install Problem
In-Reply-To: <CACL+1as-u5mcBdh2Bd_HPitMM84pKnVZAemc+tgGPe14JNHsVg@mail.gmail.com>
References: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
 <CACL+1as-u5mcBdh2Bd_HPitMM84pKnVZAemc+tgGPe14JNHsVg@mail.gmail.com>
Message-ID: <DM5PR03MB28905A15355C25A43C3066539D9D0@DM5PR03MB2890.namprd03.prod.outlook.com>

Eryk sun.  I have not been able to find the temp folder.  I am doing this on a work computer and don't have access to all the folders.  Some are hidden.

-----Original Message-----
From: eryk sun [mailto:eryksun at gmail.com] 
Sent: Tuesday, December 13, 2016 3:49 PM
To: Python-list at python.org
Cc: Rhesa Browning <rbrowning1 at mmm.com>
Subject: [EXTERNAL] Re: Install Problem

On Tue, Dec 13, 2016 at 8:37 PM, Rhesa Browning <rbrowning1 at mmm.com> wrote:
> I have been trying to install Python 3.5.2 onto my computer.  I have 
> installed and uninstalled and resinstalled several times.  Every time 
> I get an error message saying that the python35.dll doesn't exist on my computer so it can't open Python.
> How can this problem be fixed?

Please open an issue at bugs.python.org. Zip the installation logs ("Python 3.5.2*.log") from your %temp% folder, and upload them to the issue.
3M security scanners have not detected any malicious content in this message.

To report this email as SPAM, please forward it to spam at websense.com

From mok-kong.shen at t-online.de  Thu Dec 15 15:29:29 2016
From: mok-kong.shen at t-online.de (Mok-Kong Shen)
Date: Thu, 15 Dec 2016 21:29:29 +0100
Subject: A comparatively efficient software for embedding secret information
 bits into nataural language texts
Message-ID: <o2uuf3$dft$3@news.albasani.net>


WORDLISTTEXTSTEGANOGRAPHY is a new software (employing an extensive
English word list) which, while performing linguistic steganography,
also involves pseudo-random separation of the word list into two
sublists (for denoting 0 and 1 bits) that are dependent on dynamic
session-key materials, thus furnishing simultaneously substantial
cryptographical security for the embedded stego bits.

The software has a stegobit embedding rate of roughly 0.5 or higher
per word of cover-text which the user composes to be as natural as
possible under the guidance of the software. To my knowledge there
is currently no other linguistic stego software that could compete
with it, when both the naturalness of the cover texts and the
stegobit embedding rate are taken into consideration as evaluation
criteria.

The software in Python with a GUI coded in tkinter is available at:
http://s13.zetaboards.com/Crypto/topic/9024439/1/

For comments and critiques I should be very grateful.

M. K. Shen


From eryksun at gmail.com  Thu Dec 15 15:35:10 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 15 Dec 2016 20:35:10 +0000
Subject: Install Problem
In-Reply-To: <DM5PR03MB28905A15355C25A43C3066539D9D0@DM5PR03MB2890.namprd03.prod.outlook.com>
References: <DM5PR03MB289034B04B527CA8D4357B7A9D9B0@DM5PR03MB2890.namprd03.prod.outlook.com>
 <CACL+1as-u5mcBdh2Bd_HPitMM84pKnVZAemc+tgGPe14JNHsVg@mail.gmail.com>
 <DM5PR03MB28905A15355C25A43C3066539D9D0@DM5PR03MB2890.namprd03.prod.outlook.com>
Message-ID: <CACL+1auk9Crv2+zLSLM=c0A-2xzGipTfB5zizsN9BuCU31farQ@mail.gmail.com>

On Thu, Dec 15, 2016 at 8:24 PM, Rhesa Browning <rbrowning1 at mmm.com> wrote:
> I have not been able to find the temp folder.  I am doing this on a work computer
> and don't have access to all the folders.  Some are hidden.

Enter %temp% in Explorer's location bar or the Win+R dialog.


From tjreedy at udel.edu  Thu Dec 15 15:42:00 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 15:42:00 -0500
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o2uv6m$r1a$1@blaine.gmane.org>

On 12/15/2016 12:06 PM, Steve D'Aprano wrote:
> I have some key:value data where the keys often are found in contiguous
> ranges with identical values. For example:
>
> {1: "foo",
>  2: "foo",
>  3: "foo",
>  # same for keys 4 through 99
>  100: "foo",
>  101: "bar",
>  102: "bar",
>  103: "foobar",
>  104: "bar",
>  105: "foo",
> }

For this particular example (untested):

def lookup(n):
     F, B, FB = 'foo', 'bar', 'foobar'  # Ensure value objects reused
     if 1 <= n <= 100:
         return F
     elif 101 <= n <= 105:
         return (B, B, FB, B, F')[n - 101]
     else:
         raise ValueError('n must be in range(1, 106)')

> So in this case, the keys 1 through 100, plus 105, all have the same value.
>
> I have about a million or two keys, with a few hundred or perhaps a few
> thousand distinct values. The size of each contiguous group of keys with
> the same value can vary from 1 to perhaps a hundred or so.

> Has anyone dealt with data like this and could give a recommendation of the
> right data structure to use?
>
> The obvious way is to explicitly list each key, in a regular dict. But I
> wonder whether there are alternatives which may be better (faster, more
> memory efficient)?

If, as in the example, the keys comprise a contiguous sequence of ints, 
the 'obvious' way to me is sequence of values.  A tuple of constants 
gets compiled and will load quickly from a .pyc file.  4 or 8 bytes per 
entry times 1 or 2 million entries is usually tolerable on a gigabyte 
machine.

Or trade time for space with binary search or search in explicit binary 
tree.  Or combine binary search and indexed lookup as I did above.


-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Dec 15 15:48:22 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 15:48:22 -0500
Subject: Mapping with continguous ranges of keys
In-Reply-To: <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>
Message-ID: <o2uvik$qo3$1@blaine.gmane.org>

On 12/15/2016 12:27 PM, Thomas Nyberg wrote:
> On 12/15/2016 09:06 AM, Steve D'Aprano wrote:
>> Has anyone dealt with data like this and could give a recommendation
>> of the
>> right data structure to use?
>>
>
> I haven't dealt with a data structure exactly like this, but it's
> basically a sparse array.

A sparse array has at least half missing values.  This one has none on 
the defined domain, but contiguous dupicates.

 > (I'm sure it has a name somewhere in the
> academic literature, but I couldn't find it with a quick search right
> now...)

'Sparse array' is the name for sparse arrays.  I cannot think of one for 
blocks of duplicates.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Dec 15 16:09:46 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 16:09:46 -0500
Subject: =?UTF-8?Q?Re:_Python_constructors_have_particular_semantics=2c_and_?=
 =?UTF-8?B?4oCYRm9vLl9faW5pdF9f4oCZIGRvZXNuJ3QgcXVhbGlmeQ==?=
In-Reply-To: <CALwzidkQ_HryC_2zieEYM43usHHO8ReomaCMSUCVqBNVNyRktA@mail.gmail.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
 <mailman.38.1481587045.2322.python-list@python.org>
 <2311421.k3LOHGUjKi@PointedEars.de> <o2um19$l8m$1@blaine.gmane.org>
 <CALwzidkQ_HryC_2zieEYM43usHHO8ReomaCMSUCVqBNVNyRktA@mail.gmail.com>
Message-ID: <o2v0qo$ung$1@blaine.gmane.org>

On 12/15/2016 2:04 PM, Ian Kelly wrote:
> On Thu, Dec 15, 2016 at 11:05 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 12/14/2016 11:14 PM, Thomas 'PointedEars' Lahn wrote:
>>
>>> According to
>>> <https://docs.python.org/3/tutorial/classes.html#method-objects>,
>>> ?Foo.__init__? is _not_ an instance method.  Were it an instance
>>> method, the following would not happen:
>>
>>
>> This link points to subsection 9.3.4. Method Objects
>>
>>> | >>> class Foo:
>>> | ...     def __init__ (self):
>>> | ...         pass
>>> | ...
>>> | >>> Foo.__init__.__self__
>>> | Traceback (most recent call last):
>>> |   File "<stdin>", line 1, in <module>
>>> | AttributeError: 'function' object has no attribute '__self__'
>>
>>
>> You have misread the docs.  Foo.__init__ is the function. Foo().__init__ is
>> a method object with the attribute __self__.  You omitted the ()s.
>
> I think this is actually the point that Thomas was making. He was
> responding to the assertion that "Foo.__init__" is an instance method
> and demonstrating that it's false because Foo is the class, not an
> instance of Foo.

The __init__ function *is* an 'instance method', but not a (bound) 
method object.  The distinction is important.  I admit that 'instance 
method' can be confusing if one does not understand the contextual 
meaning of the two words.  A 'method' is a function that is a class 
attribute. Being a method normally adds special behavior to the 
function.  An 'instance method', the default status of a method,  takes 
an *instance of the class* (or subclass) as first parameter.  A 'class 
method' (relatively rare) takes the *class* (or subclass) as first 
parameter.   A 'static method' (very rare) takes neither as first 
parameter and is really just a function.

-- 
Terry Jan Reedy




From tomuxiong at gmx.com  Thu Dec 15 16:30:20 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Thu, 15 Dec 2016 13:30:20 -0800
Subject: Mapping with continguous ranges of keys
In-Reply-To: <o2uvik$qo3$1@blaine.gmane.org>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>
 <o2uvik$qo3$1@blaine.gmane.org>
Message-ID: <90e54769-94c8-abd3-95a5-4872cb12e86e@gmx.com>

On 12/15/2016 12:48 PM, Terry Reedy wrote:
> On 12/15/2016 12:27 PM, Thomas Nyberg wrote:
>>
>> I haven't dealt with a data structure exactly like this, but it's
>> basically a sparse array.
>
> A sparse array has at least half missing values.  This one has none on
> the defined domain, but contiguous dupicates.
>

I'm sorry for devolving into semantics, but there certainly isn't a 
single definition of "sparse array" out there. For example, the 
definition in wikipedia (https://en.wikipedia.org/wiki/Sparse_array) 
doesn't agree with you:

"In computer science, a sparse array is an array in which most of the 
elements have the default value (usually 0 or null)."

Personally my usage of sparse arrays in scipy has _always_ had all 
defined values it's just that the default value was 0. I never deal with 
"missing" values.

>> (I'm sure it has a name somewhere in the
>> academic literature, but I couldn't find it with a quick search right
>> now...)
>
> 'Sparse array' is the name for sparse arrays.  I cannot think of one for
> blocks of duplicates.
>

Yeah that's why I said "basically a sparse array". It has the same 
defining characteristics. It is a situation where you have a large 
number of values associated to a small number of values in such a way 
that you can fairly easily describe the association without needing to 
simply write out all pairs. For regular sparse arrays it's easy because 
you associate them to a single value by default and only specify the 
ones that aren't. In this case that doesn't work, but due to the fact 
that you have long runs of repeated values you can use that to encode 
and compress the mapping.

Still not sure what to call it. I like D'Arcy's idea of subclassing a 
dict in combination with Peter's idea of using bisect (I had no idea 
that module existed!) so maybe I'll change my own not so great 
terminology to "basically a sparse map".

Cheers,
Thomas


From tjreedy at udel.edu  Thu Dec 15 16:57:24 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 15 Dec 2016 16:57:24 -0500
Subject: Unicode script
In-Reply-To: <392bb301-0e54-3607-7b7f-3c13e7bb6b9b@mrabarnett.plus.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
 <392bb301-0e54-3607-7b7f-3c13e7bb6b9b@mrabarnett.plus.com>
Message-ID: <o2v3k1$tqs$1@blaine.gmane.org>

On 12/15/2016 1:06 PM, MRAB wrote:
> On 2016-12-15 16:53, Steve D'Aprano wrote:
>> Suppose I have a Unicode character, and I want to determine the script or
>> scripts it belongs to.
>>
>> For example:
>>
>> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
>> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
>> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>>
>>
>> Is this information available from Python?
>>
>>
>> More about Unicode scripts:
>>
>> http://www.unicode.org/reports/tr24/
>> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
>> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt
>>
>>
> Interestingly, there's issue 6331 "Add unicode script info to the
> unicode database". Looks like it didn't make it into Python 3.6.

https://bugs.python.org/issue6331
Opened in 2009 with patch and 2 revisions for 2.x.  At least the Python 
code needs to be updated.

Approved in principle by Martin, then unicodedata curator, but no longer 
active.  Neither, very much, are the other 2 listed in the Expert's index.

 From what I could see, both the Python API (there is no doc patch yet) 
and internal implementation need more work.  If I were to get involved, 
I would look at the APIs of PyICU (see Eryk Sun's post) and the 
unicodescript module on PyPI (mention by Pander Musubi, on the issue).

-- 
Terry Jan Reedy




From steve+python at pearwood.info  Thu Dec 15 18:28:55 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 16 Dec 2016 10:28:55 +1100
Subject: Python constructors have particular semantics2c and
 =?UTF-8?B?4oCYRm9vLl9faW5pdF9f4oCZ?= doesn't qualify
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <851sxc1z1m.fsf_-_@benfinney.id.au>
 <mailman.38.1481587045.2322.python-list@python.org>
 <2311421.k3LOHGUjKi@PointedEars.de> <o2um19$l8m$1@blaine.gmane.org>
 <CALwzidkQ_HryC_2zieEYM43usHHO8ReomaCMSUCVqBNVNyRktA@mail.gmail.com>
 <o2v0qo$ung$1@blaine.gmane.org>
 <mailman.110.1481836199.2322.python-list@python.org>
Message-ID: <58532739$0$1601$c3e8da3$5496439d@news.astraweb.com>

On Fri, 16 Dec 2016 08:09 am, Terry Reedy wrote:

> The __init__ function *is* an 'instance method', but not a (bound)
> method object.  The distinction is important.  I admit that 'instance
> method' can be confusing if one does not understand the contextual
> meaning of the two words.  A 'method' is a function that is a class
> attribute. Being a method normally adds special behavior to the
> function.  An 'instance method', the default status of a method,  takes
> an *instance of the class* (or subclass) as first parameter.  A 'class
> method' (relatively rare) takes the *class* (or subclass) as first
> parameter.   A 'static method' (very rare) takes neither as first
> parameter and is really just a function.

I think that what Terry might be trying to get at is that there's actually
two distinct but related meanings for "instance method" in the context of
Python. (Other languages may be different.)

(Terry, forgive me if I'm misinterpreting what you are saying.)

The first is an object of type types.MethodType. Because functions and
methods are first-class values in Python, such instance methods could be
stuffed into lists, attached as attributes to unrelated objects, bound to
variables, etc. Instance methods are just a data type, and could come from
anywhere.

Proof of concept:

py> class X(object):
...     pass
...
py> x = X()
py> from types import MethodType
py> def method(self):
...     print("method bound to", self)
...
py> x.surprise = MethodType(method, "Hello world!")
py> x.surprise()
method bound to Hello world!


But the usual meaning of "instance method" is a function defined inside a
class body (with exceptions below):

class X(object):
    def method(self):
        ...


That is *conceptually* an instance method even if the implementation uses an
object of different type. In Python 2, retrieving "method" from either the
class or the instance returns something which matches both definitions of
object:

X.method returns an "unbound method", an object of type MethodType;
X().method returns a "bound method", also an object of type MethodType.


But in Python 3, it is recognised that there is little or no conceptual
difference between an unbound method and a function, and the implementation
is changed:

X.method returns an "unbound method", an object of type FunctionType;
X().method returns a "bound method", an object of type MethodType.


But whether the implementation is a special form of MethodType or just
FunctionType, the object is still conceptually an instance method.


The exceptions to the "function defined inside a class body" rule includes
two built-in types used as decorators:

classmethod
staticmethod

which are deemed to be "kinds of methods" but not *instance* methods. In
Python, we call them "class methods" and "static methods", but in other
languages they may not exist at all or have other names.

(In Java, the closest equivalent to "class method" is called a "static
method".)



Adding to the complexity, the __name__ of MethodType is just "method", and
the __name__ of FunctionType is just "function", so introspecting the
objects (looking at their repr() etc) may give slightly different results.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From python at lucidity.plus.com  Thu Dec 15 19:18:06 2016
From: python at lucidity.plus.com (Erik)
Date: Fri, 16 Dec 2016 00:18:06 +0000
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <eb9hu6Fjaf6U1@mid.individual.net>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <eb8j88Fco3aU1@mid.individual.net>
 <5e175c03-4afe-4c46-a3fe-86dd3ef6f583@googlegroups.com>
 <eb9hu6Fjaf6U1@mid.individual.net>
Message-ID: <280ae740-1e7b-7fa6-0fae-8bd91af76243@lucidity.plus.com>

On 13/12/16 06:14, Gregory Ewing wrote:
> Ned Batchelder wrote:
>> if a C++ constructor raises an exception, will the corresponding
>> destructor
>> be run, or not? (No, because it never finished making an object of
>> type T.)
>
> So it just leaks any memory that's been allocated by
> the partially-run constructor?

If you're referring to resources in general that a constructor (in the 
C++ sense) allocates - memory, file descriptors, whatever - then it's up 
to the constructor to release those resources before throwing the 
exception should something fail. Destructors are not executed for 
objects that were not constructed.

If the constructor is not written to that standard, then yes - it will 
leak resources.

The memory allocated for the _object_ itself will be released though.

Regards, E.


From python at lucidity.plus.com  Thu Dec 15 19:36:46 2016
From: python at lucidity.plus.com (Erik)
Date: Fri, 16 Dec 2016 00:36:46 +0000
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
Message-ID: <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>

On 12/12/16 23:23, Chris Angelico wrote:
> In JavaScript, it's normal to talk about "calling a function as a
> constructor". When you do, there is a 'this' object before you start.

No there isn't. There is an implicit binding of a variable called "this" 
based on the syntactic sugar of whether you're calling a function as 
method on an object or not.

In "strict" mode, this has been redefined to be "undefined" (i.e., there 
is no object) for when you're not - otherwise it will be a binding to 
the global "document" object (and in Node.js, I think something else 
entirely. It's a mess ...).

> Ultimately, every language has slightly different semantics

You're not wrong ;)

E.



From rosuav at gmail.com  Thu Dec 15 20:17:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 16 Dec 2016 12:17:28 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
Message-ID: <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>

On Fri, Dec 16, 2016 at 11:36 AM, Erik <python at lucidity.plus.com> wrote:
> On 12/12/16 23:23, Chris Angelico wrote:
>>
>> In JavaScript, it's normal to talk about "calling a function as a
>> constructor". When you do, there is a 'this' object before you start.
>
>
> No there isn't. There is an implicit binding of a variable called "this"
> based on the syntactic sugar of whether you're calling a function as method
> on an object or not.
>
> In "strict" mode, this has been redefined to be "undefined" (i.e., there is
> no object) for when you're not - otherwise it will be a binding to the
> global "document" object (and in Node.js, I think something else entirely.
> It's a mess ...).

I'm talking about when you call a function as a constructor: "new
Foo()". Doesn't that have a 'this' object before the function starts?

ChrisA


From python at mrabarnett.plus.com  Thu Dec 15 21:44:20 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 16 Dec 2016 02:44:20 +0000
Subject: Unicode script
In-Reply-To: <o2v3k1$tqs$1@blaine.gmane.org>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
 <392bb301-0e54-3607-7b7f-3c13e7bb6b9b@mrabarnett.plus.com>
 <o2v3k1$tqs$1@blaine.gmane.org>
Message-ID: <c321b2ad-0615-212c-70d5-30f039d8765b@mrabarnett.plus.com>

On 2016-12-15 21:57, Terry Reedy wrote:
> On 12/15/2016 1:06 PM, MRAB wrote:
>> On 2016-12-15 16:53, Steve D'Aprano wrote:
>>> Suppose I have a Unicode character, and I want to determine the script or
>>> scripts it belongs to.
>>>
>>> For example:
>>>
>>> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
>>> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
>>> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>>>
>>>
>>> Is this information available from Python?
>>>
>>>
>>> More about Unicode scripts:
>>>
>>> http://www.unicode.org/reports/tr24/
>>> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
>>> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt
>>>
>>>
>> Interestingly, there's issue 6331 "Add unicode script info to the
>> unicode database". Looks like it didn't make it into Python 3.6.
>
> https://bugs.python.org/issue6331
> Opened in 2009 with patch and 2 revisions for 2.x.  At least the Python
> code needs to be updated.
>
> Approved in principle by Martin, then unicodedata curator, but no longer
> active.  Neither, very much, are the other 2 listed in the Expert's index.
>
>  From what I could see, both the Python API (there is no doc patch yet)
> and internal implementation need more work.  If I were to get involved,
> I would look at the APIs of PyICU (see Eryk Sun's post) and the
> unicodescript module on PyPI (mention by Pander Musubi, on the issue).
>
For what it's worth, the post has prompted me to get back to a module I 
started which will report such Unicode properties, essentially the ones 
that the regex module supports. It just needs a few more tweaks and 
packaging up...



From metal.suomi at gmail.com  Thu Dec 15 23:29:24 2016
From: metal.suomi at gmail.com (metal.suomi at gmail.com)
Date: Thu, 15 Dec 2016 20:29:24 -0800 (PST)
Subject: just started
Message-ID: <7e1190f3-e5be-4a39-ad68-35abb9832653@googlegroups.com>

Hi everybody, I have just installed python 3.5.2 (downloaded from https://www.python.org/) on my mac (El Capitan). 

1) I see from command line that older version of python are already available on my machine (python, python2.6, python2.7, python3, python3.5). I guess some might have been installed as part of Xcode (?) or via macport (? despite I never explicitly asked for python).

2) if I wish to install extra libraries, in particularly SciPy, what's best way of doing it? From command line I see the following commands: pip3 and pip3.5. I guess pip3.5 is for python3.5 ? Can I use pip3 and pip3.5 interchangeably, or pip3 will only install things for python3, and not 3.5?

Are all these command equivalent?

pip3 install --user numpy scipy matplotlib
pip3.5 install --user numpy scipy matplotlib

python3.5 -m pip install numpy scipy matplotlib


3) I also see this command to install it from macport, although not sure it will be ok given I didn't install python3.5 via macport but from downaloaded package?

sudo port install py35-numpy py35-scipy py35-matplotlib py35-ipython +notebook py35-pandas py35-sympy py35-nose

Thanks! I'm going through a very confused start, with too many version and too many commands to guess the right way of doing.

M


From rosuav at gmail.com  Thu Dec 15 23:55:53 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 16 Dec 2016 15:55:53 +1100
Subject: just started
In-Reply-To: <7e1190f3-e5be-4a39-ad68-35abb9832653@googlegroups.com>
References: <7e1190f3-e5be-4a39-ad68-35abb9832653@googlegroups.com>
Message-ID: <CAPTjJmqumgxdPunBnQkL6C93i9-ZOAqDB-gOMyJ_Oqqjvcd7Yg@mail.gmail.com>

On Fri, Dec 16, 2016 at 3:29 PM,  <metal.suomi at gmail.com> wrote:
> 2) if I wish to install extra libraries, in particularly SciPy, what's best way of doing it? From command line I see the following commands: pip3 and pip3.5. I guess pip3.5 is for python3.5 ? Can I use pip3 and pip3.5 interchangeably, or pip3 will only install things for python3, and not 3.5?
>
> Are all these command equivalent?
>
> pip3 install --user numpy scipy matplotlib
> pip3.5 install --user numpy scipy matplotlib
>
> python3.5 -m pip install numpy scipy matplotlib

In theory, "pip3" will install into the default "python3", whichever
that is. However, in practice, it's entirely possible that it installs
into a very different Python from the one you're expecting. The most
reliable form is the latter; whatever command you use to start Python,
add "-m pip" to it, and you know you're talking to that same
installation.

ChrisA


From steve+python at pearwood.info  Fri Dec 16 00:38:20 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 16 Dec 2016 16:38:20 +1100
Subject: A routine to calc Shannon's info entropy. I'm open to
 improvements/suggestions.
References: <o2v4vi$h3b$3@dont-email.me>
Message-ID: <58537dce$0$1609$c3e8da3$5496439d@news.astraweb.com>

On Fri, 16 Dec 2016 09:21 am, DFS wrote:

> Code is based on the formula and results here:
> http://www.shannonentropy.netmark.pl
[...]
> Seems to work fine.
> 
> Any suggestions to improve the code?


- Separate the calculation logic from the display logic as much
  as practical.

- Modular programming: use functions to make it easier to test and easier
  to modify the program in the future, e.g. to add command line options.

- Use standard tools where possible.

- Better, more descriptive names.

- Avoid global variables unless really needed.

- More error checking.

- Errors should exit with a non-zero return code, and should print to
  stderr rather than stdout.

You end up with about twice as much code, but hopefully it is easier to
understand, and it should certainly be easier to debug and maintain if you
decide to change it.



# --- cut ---

# only needed in Python 2
from __future__ import division

import sys
import math
import string

from collections import Counter


def fatal_error(errmsg):
    # Python 3 syntax
    print(errmsg, file=sys.stderr)
    ## Python 2 syntax
    ## print >>sys.stderr, errmsg
    sys.exit(1)


def display_header(msg):
    print()
    print("%s characters in '%s'" % (len(msg), msg))
    print()
    print(" #   Char  Freq   Dist    D*log2(D)  H(X) sum")
    print("---  ----  ----  ------  ----------  ----------")


def display_row(row_number, c, freq, dist, entropy, running_total):
    args = (row_number, c, freq, dist, entropy, running_total)
    template = "%3d  %-4c  %4d  %6.3f  %10.3f  %10.5f"
    print(template % args)


def entropy(c, freqs, num_symbols):
    """Return the entropy of character c from frequency table freqs."""
    f = freqs[c]/num_symbols
    return -f*math.log(f, 2)


def display_results(freqs, Hs, num_symbols):
    """Display results including entropy of each symbol.

    Returns the total entropy of the message.
    """
    # Display rows with uppercase first, then lower, then digits.
    upper = sorted(filter(str.isupper, freqs))
    lower = sorted(filter(str.islower, freqs))
    digits = sorted(filter(str.isdigit, freqs))
    assert set(upper + lower + digits) == set(freqs)
    count = 1
    running_total = 0.0
    for chars in (upper, lower, digits):
        for c in chars:
            f = freqs[c]
            H = Hs[c]
            running_total += H
            display_row(count, c, f, f/num_symbols, H, running_total)
            count += 1
    total = running_total
    print()
    print("The Shannon entropy of your message is %.5f" % total)
    print("The metric entropy of your message is %.5f"% (total/num_symbols))
    return total


def main(args=None):
    if args is None:
        args = sys.argv[1:]
    if len(args) != 1:
        fatal_error("too many or too few arguments")
    msg = args[0]
    if not msg.isalnum():
        fatal_error("only alphanumeric symbols supported")
    display_header(msg)
    frequencies = Counter(msg)
    num_symbols = len(msg)
    # Calculate the entropy of each symbol and the total entropy.
    entropies = {}
    for c in frequencies:
        H = entropy(c, frequencies, num_symbols)
        entropies[c] = H
    total = display_results(frequencies, entropies, num_symbols)



if __name__ == "__main__":
    # Only run when module is being used as a script.
    main()


# --- cut ---





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From tjreedy at udel.edu  Fri Dec 16 02:57:40 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 16 Dec 2016 02:57:40 -0500
Subject: Mapping with continguous ranges of keys
In-Reply-To: <90e54769-94c8-abd3-95a5-4872cb12e86e@gmx.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>
 <o2uvik$qo3$1@blaine.gmane.org>
 <90e54769-94c8-abd3-95a5-4872cb12e86e@gmx.com>
Message-ID: <o306ph$vt6$1@blaine.gmane.org>

On 12/15/2016 4:30 PM, Thomas Nyberg wrote:
> On 12/15/2016 12:48 PM, Terry Reedy wrote:
>> On 12/15/2016 12:27 PM, Thomas Nyberg wrote:
>>>
>>> I haven't dealt with a data structure exactly like this, but it's
>>> basically a sparse array.
>>
>> A sparse array has at least half missing values.  This one has none on
>> the defined domain, but contiguous dupicates.
>>
>
> I'm sorry for devolving into semantics, but there certainly isn't a
> single definition of "sparse array" out there. For example, the
> definition in wikipedia (https://en.wikipedia.org/wiki/Sparse_array)
> doesn't agree with you:

I think it does ;-).

> "In computer science, a sparse array is an array in which most of the
> elements have the default value (usually 0 or null)."

Let's devolve to a memory-based language like C. An physical array 
consisting of sequential memory locations must have *some* bit pattern 
in every byte and hence in every multibyte block.  If I remember 
correctly, C malloc initialized bytes to all 0 bits, which is an int 
value of 0 also.  If there is no meaningful value, there must be a 
default value that means 'missing'.  0 may mean 'missing'. Null values 
are by definition non-values.  Python uses None as a Null.  If the 
example had been populated largely with Nones, I would have called it 
'sparse'.

 > Personally my usage of sparse arrays in scipy has _always_ had all
 > defined values it's just that the default value was 0. I never deal
 > with "missing" values.

Lucky you.  In statistics and analysis of real, experimental data, 
missing values are often a possibility.  They are always a nuisance, 
sometimes a major one.  When the data are represented by arrays,  rather 
than by some compacted form, some value has to be chosen to represent 
the absence of of data.

-- 
Terry Jan Reedy



From skybuck2000 at hotmail.com  Fri Dec 16 06:59:04 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Fri, 16 Dec 2016 03:59:04 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
Message-ID: <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>

Hi thanks for your (Dennis) reply,

You seem to have two main questions:

1. Is the attack order sequential in time or is it positional ?

The answer to this question is kinda:

Both, (which is kinda funny, since you did not think of this possibility that it could be both, which you also did with the other issue, see below ;))

It indicates which object should be attacked first. And ofcourse objects have a position in the real world, so this also implies position to some degree.

2. Do the numbers mean types or instances ?

Again the answer to this question is kinda:

Both, (^<-which is the real funny one, because the original model where there were only 4 types and 7 instances was still too large to compute within reasonable time. So the number of instances has been brought back to the number of types to represent each type exactly once so to at least keep the model somewhat realistic and include all types which is kinda important. I am particularly interested in type 4 so it has to be in the model :) Type 4 can attack the other 3 types with easy this gives you some hint at what kind of type it might be ;)).

(So there are 4 types, and 4 instances, which happen to overlap each other, so you can consider them the same thing type=instance (to simplify the program)).

These were your two most important questions, however you seem to have some additional (sub) questions which I will also try and answer.


3. One of your questions was: why are the indexes called "index1,index2" and so forth. This is simply because there are 8 objects. Each object needs it's own index to create a brute force algorithm which can create all combinations of the object attack patterns. The code for this is pretty simple/straight forward so I will mention it below, I will also rename these indexes as you requested to give them more meaning, pascal style:

for FriendlyIndex1 := 1 to 24 do
    for FriendlyIndex2 := 1 to 24 do
        for FriendlyIndex3 := 1 to 24 do
            for FriendlyIndex4 := 1 to 24 do
                for EnemyIndex1 := 1 to 24 do
                    for EnemyIndex2 := 1 to 24 do
                        for EnemyIndex3 := 1 to 24 do
                            for EnemyIndex4 := 1 to 24 do
                                // ComputeCombat( FriendlyIndex1,FriendlyIndex2,FriendlyIndex3,FriendlyIndex4, EnemyIndex1,EnemyIndex2,EnemyIndex3,EnemyIndex4)

So these indexes are simply 8 integer variables used to generate all possible attack orders (permutation number).

Each index can then be used to look up the actual permutation and used in combat...

So this iteration code is a big help to make sure and produce all combinations, it's simple, it's effective, it's clear to what it's doing.

ComputeCombat could be a routine or simply replaced with actual code to prevent "variable sharing issues". My suggestion in case you ever do try to write code for it is to keep everything "global" or simply inside a single routine... so
that parameter hell or whatever doesn't occur... keep it as simple as possible for a first version... then later it can be enhance with nice python features. Ofcourse if you think the problem is simple enough to try using more advanced features at first you welcome to try that as well but I would find it very funny if that fails... so when it does fail... fall back to ultra-simple code :) Perhaps python doesn't even have ultra simple data structures which might actually complexify your capability of solving this problem, which would be somewhat of an interesting conclusion regarding the python language as a whole ! Consider yourself challenged by me to solve it and prove that Python is not a bad language ! LOL :) Yes little bit trollish but for good reason. Eventually I do suspect python might be of some help... at least you mentioned it can generate permutations... but that is a sub problem in this case...

4. There was one more somewhat interesting sub question, your question seems to be about the attack/victory table, you seem to wonder about symetrical/asymetrical, and why it would not be symetrical ?

I think I can explain this issue for you. The reason why it's not symetrical is that tanks for example have different armor thickness depending on their angle. So let's say Tank X sneaks up on Tank Y and Tank X manages to hit the Tank X in the back... then Tank X's victory chance is much higher then if Tank X was sneaked up on by Tank Y... in that case Tank X's victory chance would be much lower.

I think this is the main reason why you are seeing an asymterical victory chance table. I hope that clears it up for you ;)

I think I have solved all your confusion, you should now have enough information to write a computer program that could solve the problem. Solving the problem entirely would not be necessary for you to share any possible python enhancements or pseudo code or own techniques or (computing efficiency) ideas you would have come up with it. Just compiling it and running it a little bit should be sufficient to see if it actually works.

Bye,
  Skybuck.


From skybuck2000 at hotmail.com  Fri Dec 16 07:31:31 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Fri, 16 Dec 2016 04:31:31 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
Message-ID: <408e8327-908c-4892-ae69-8f3548edbdff@googlegroups.com>

A nice chinese-like saying might have come out of this, perhaps it even already exists:

"When you confused, the answer might be fused !" :)

Sounds like TZen Su ! That dude from Art of War or something like that :)


I will take this chance to correct a little pretty obvious typo corrected:

(*) Tank X changed to Tank Y

So let's say Tank X sneaks up on Tank Y and Tank X manages to hit the (*) Tank Y in the back... then Tank X's victory chance is much higher then if Tank X was sneaked up on by Tank Y... in that case Tank X's victory chance would be much lower.


From m at funkyhat.org  Fri Dec 16 09:03:02 2016
From: m at funkyhat.org (Matt Wheeler)
Date: Fri, 16 Dec 2016 14:03:02 +0000
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwc8brRcGE2fotW5-541YfzxgMe3H2FzwvbUPA3Nz40-w@mail.gmail.com>
 <CANc-5Uy+D7dqfjMGZhovQ1aoH8RT1ENvDqEb4SXcrgNoHRwUEA@mail.gmail.com>
 <CANc-5UwXQDQ4c4TekZO9aYEL_fudm7iGziws52AkQHX4s14qHQ@mail.gmail.com>
 <CANc-5Ux6m9hJDeYEZ4V3ExHR=XtTmywmVvaPYY3s9w3s3UqqHA@mail.gmail.com>
 <CANc-5UyAhmA4Oq1C3Rk+aebCB35-h17SbhAC2_YwmH1NNvzF_Q@mail.gmail.com>
 <CANc-5Ux+x6cQnM+7b71RgYN8P1o3RarF53o_0K6TG1sf+5oBjQ@mail.gmail.com>
 <CANc-5UzTO_zaUQeTW0sBtc4_rU1r=JeVQ0SiH-0zsrxnotjqtQ@mail.gmail.com>
 <CANc-5UyPhS+gD-BuxWvQNDw7Wk-VsAa8fK7xMU9Osao9BVdZVQ@mail.gmail.com>
 <CANc-5UxgY3vL1VprybVXCxF_5YgL0cjVDFY_br1TS5nuvxEOkA@mail.gmail.com>
 <CANc-5Uwi+Vp+inGKooN7GSbk=hppQYWOjSNCrbb6EVNgmh+UgA@mail.gmail.com>
 <CANc-5Uy7-aA32y1gLcp=zLnHc-itTb4xOAikqHxgO4mDwx6DcA@mail.gmail.com>
 <CANc-5UyOQopd75p9K6fWmOmtQcdW0Td-O-6R=_ZFVaBQvcKqTg@mail.gmail.com>
 <CANc-5UzaLXqUWUro6nJ-x8VmCtaeCWrwEayGE0X4i9ZtAm0QSw@mail.gmail.com>
 <CANc-5UzMN19xyY07q3DgKXyR9M=C4CYAUhQoXrhL6RuUGMPS=A@mail.gmail.com>
 <CANc-5Uw2ta34basMuG88S4bGG=sGSVwxK4gzm=qAy0DHK-CGDQ@mail.gmail.com>
 <CANc-5UxvefxVYoZqNVVBbEYWbpjnbGk-a964_6Hae2WFNwj7_w@mail.gmail.com>
 <CANc-5Uxw=AScfO0+LVUgL9X29JgH9cqVrJj8=dKd6Q0tc7gLsg@mail.gmail.com>
 <CANc-5Uy1usatpzb3Aq0616TEuQhx93QNgF_xkhboSaPX4E5nVQ@mail.gmail.com>
 <CANc-5UwgHQvqeM5GspNrAEfGu6U2hgry3mmA-ys8Oz9Rs47oiw@mail.gmail.com>
 <CANc-5Uz6o5Cj0U=Nd+Zjihd-NLevV1guo4nUdqx6-5TM4O0QoQ@mail.gmail.com>
 <CANc-5UzE_vmgN+BOz8sFHJuxxpzXmhVf3nVocLvLg1GnRV1zMg@mail.gmail.com>
 <CANc-5UxixW1mKQXmm56d_eGgKcpz0tLkNqdTnC36p3uLUoN+TQ@mail.gmail.com>
 <CANc-5UxhuK6gjurgSPKf15=YWv1c4vQxEeCSksz3q08huXMSVQ@mail.gmail.com>
 <CANc-5UznaAoZZZWL_W-nrLr+=2+QsrNPFHmg_y75SA8h_LUX_Q@mail.gmail.com>
 <CANc-5Uw7oNghOR+jFbtoDcsa7cFDR3B+Q_ZySDTLRN9gmL4aQw@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
Message-ID: <CAG93HwGYK5Ms8qyLq9ym=K7sjk4AsgLNjGKqqmwcFD=mDiot6Q@mail.gmail.com>

On Wed, 14 Dec 2016, 01:06 Skip Montanaro, <skip.montanaro at gmail.com> wrote:

I know this isn't a Python-specific question, but i got zero useful
responses from the help-gnu-emacs list for some reason. I think this
expression should not evaluate to the empty set:

set(python_programmers) & set(emacs_users) & set(new_macbookpro_owners)


So I fit 2 of these, but am a vim user rather than emacs.

Hopefully there are a few people out there who've had an opportunity to try
the new ESC-less Pros with the Touch Bar for an extended period of time.
Does the lack of a physical ESC key create problems for people, especially
Emacs users?


No. I think as long as you're happy with the keyboard the esc key is fine.
I happen to like short travel keyboards, and while the lack of physical
feedback from the touch bar when hitting esc was slightly jarring initially
I got used to it within a day.
That short travel is worth bearing in mind in general though, I know a lot
of people say they don't like that.

Just to head off a couple suggestions people might be inclined to make...
Yes, I know I can use C-[ or the Alt key instead of ESC. I can remap other
keys like Caps Lock or the back tick. I can also buy some other laptop with
a true ESC key, or buy a 13-inch MBP sans Touch Bar. I do plan to try out


The 13" without touch bar only has 2 usb-c ports, which I know is a whole
other discussion but I've found the 2 ports on a side of the 15" a bit too
cramped for some of my accessories.

Emacs on a Touch-Bar-equipped MacBook Pro at the Apple Store, but a few
minutes horsing around in the din of holiday shopping isn't the same as an
extended test drive in my usual environment.

So, for those of you who've tried it, does the lack of a physical ESC key
create problems?


So... Not for me, but obviously with the caveats above. Giving it a try in
an Apple store is definitely a good idea :)

-- 

--
Matt Wheeler
http://funkyh.at


From steve+python at pearwood.info  Fri Dec 16 09:27:43 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 17 Dec 2016 01:27:43 +1100
Subject: Mapping with continguous ranges of keys
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <5853f9e0$0$1610$c3e8da3$5496439d@news.astraweb.com>

On Fri, 16 Dec 2016 04:06 am, Steve D'Aprano wrote:

> I have some key:value data where the keys often are found in contiguous
> ranges with identical values.
[...]


Thank you to everyone who gave suggestions!

I have experimented with two solutions, and hope somebody might be able to
suggest some improvements. Attached is the test code I ran, suggestions for
improving the performance will be appreciated.

I decided on these two data structures:

(1) The naive solution: don't worry about duplicate values, or the fact that
keys come in contiguous groups, and just store each individual key and
value in a dict; this is faster but uses more memory.

(2) The clever solution: use a pair of lists, one holding the starting value
of each group of keys, and the other holding the common values. This saves
a lot of memory, but is slower.

A concrete example might help. Suppose I have 15 keys in five groups:

D = {0: 10,
     1: 20, 2: 20,
     3: 30, 4: 30, 5: 30,
     6: 40, 7: 40, 8: 40, 9: 40,
     10: 50, 11: 50, 12: 50, 13: 50, 14: 50}

(Remember, in reality I could have as many as a million or two keys. This is
just a simple toy example.)

Instead of using a dict, I also set up a pair of lists:

L = [0, 1, 3, 6, 10, 15]  # starting value of each group
V = [10, 20, 30, 40, 50]  # value associated with each group

Note that the first list has one extra item, namely the number one past the
final group.

I can do a key look-up using either of these:

    D[key]

    V[bisect.bisect_right(L, i) - 1]


I tested the memory consumption and speed of these two solutions with
(approximately) one million keys. I found:

- the dict solution uses a lot more memory, about 24 bytes per key, compared
to the pair of lists solution, which is about 0.3 bytes per key;

- but on my computer, dict lookups are about 3-4 times faster.


Any suggestions for improving the speed of the binary search version, or the
memory consumption of the dict?

By the way: the particular pattern of groups in the sample code (groups of
size 1, 2, 3, ... up to 50, then repeating from 1, 2, 3, ... again) is just
demo. In my real data, the sizes of the groups are all over the place, in
an unpredictable pattern.



Thanks in advance.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.


From stephane at wirtel.be  Fri Dec 16 09:49:33 2016
From: stephane at wirtel.be (Stephane Wirtel)
Date: Fri, 16 Dec 2016 15:49:33 +0100
Subject: Last call for the Call For Proposals of PythonFOSDEM 2017
Message-ID: <20161216144933.qquqd4ljjpqkuw3p@sg1>

Hello, this week-end is the last two days for the Call For Proposals of 
PythonFOSDEM 2017. We have received a lot of topics, but if you want to 
become a speaker and that you have a very cool topic to submit, please 
don't hesite and send us your proposal.

Deadline is 2016-12-18.

Stephane


Call For Proposals
==================

This is the official call for sessions for the Python devroom at FOSDEM 2017.

FOSDEM is the Free and Open source Software Developers' European Meeting, a free
and non-commercial two-day week-end that offers open source contributors a place
to meet, share ideas and collaborate.

It's the biggest event in Europe with +5000 hackers, +400 speakers.

For this edition, Python will be represented by its Community.
If you want to discuss with a lot of Python Users, it's the place to be!

Important dates
===============

* Submission deadlines: 2016-12-18
* Acceptance notifications: 2016-12-23

Practical
=========

* The duration for talks will be 30 minutes, including presentations and
questions and answers.
* Presentation can be recorded and streamed, sending your proposal implies
giving permission to be recorded.
* A mailing list for the Python devroom is available for discussions about
devroom organisation. You can register at this address:
https://lists.fosdem.org/listinfo/python-devroom

How to submit
=============

All submissions are made in the Pentabarf event planning tool at
https://penta.fosdem.org/submission/FOSDEM17

When submitting your talk in Pentabarf, make sure to select the Python devroom
as the Track.

Of course, if you already have a user account, please reuse it.

Questions
=========

Any questions, please send an email to info AT python-fosdem DOT org

Thank you for submitting your sessions and see you soon in Brussels to talk
about Python.

If you want to keep informed for this edition, you can follow our twitter
account @PythonFOSDEM.

* FOSDEM 2017: https://fosdem.org/2017
* Python Devroom: http://python-fosdem.org
* Twitter: https://twitter.com/PythonFOSDEM


Stephane

-- 
St?phane Wirtel - http://wirtel.be - @matrixise


From grant.b.edwards at gmail.com  Fri Dec 16 10:11:54 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 16 Dec 2016 15:11:54 +0000 (UTC)
Subject: Reading python list as a newsgroup (was ...)
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com>
Message-ID: <o3107q$gm5$1@blaine.gmane.org>

On 2016-12-16, Dennis Lee Bieber <wlfraed at ix.netcom.com> wrote:
> On Thu, 15 Dec 2016 13:34:32 -0500, Terry Reedy <tjreedy at udel.edu> declaimed the following:
>>
>>If you want to read python-list as a news group, you might try 
>>news.gmane.org.  About once a year, it goes down for a few hours to a 
>>day, but has otherwise been dependable.  I found it when my ISP dropped 
>>newsgroup access around a decade ago.
>
> And I found it when the spam on comp.lang.python got overly
> annoying.

I didn't notice much spam on c.l.p (but then again, I filter out all
posts from google-groups).  The problem on c.l.p that caused me to
switch to gmane's NNTP server was the number of broken references.
They both have references that break in various scenarios, but my
tests indicated that it was worse on c.l.p.  [Yes, I actually wrote a
Python program that used an NNTP client library to check all the
reference values in a large sample of posts from both sources.]

-- 
Grant Edwards               grant.b.edwards        Yow! I wonder if I could
                                  at               ever get started in the
                              gmail.com            credit world?



From darcy at vex.net  Fri Dec 16 11:12:11 2016
From: darcy at vex.net (D'Arcy Cain)
Date: Fri, 16 Dec 2016 11:12:11 -0500
Subject: Reading python list as a newsgroup (was ...)
In-Reply-To: <o3107q$gm5$1@blaine.gmane.org>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com> <o3107q$gm5$1@blaine.gmane.org>
Message-ID: <122fd551-8bd2-dd24-3bda-41de8e5798ca@vex.net>

On 2016-12-16 10:11 AM, Grant Edwards wrote:
> I didn't notice much spam on c.l.p (but then again, I filter out all
> posts from google-groups).  The problem on c.l.p that caused me to

Yes, blocking GG was the biggest improvement to my reading this list 
(mailing list in my case).  That and a few judicious PLONKs and it is 
now useful.

Al I need to do now is figure out how to filter out the almost daily 
Windows installer questions.

-- 
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:darcy at Vex.Net
VoIP: sip:darcy at Vex.Net


From tomuxiong at gmx.com  Fri Dec 16 13:04:05 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Fri, 16 Dec 2016 10:04:05 -0800
Subject: Mapping with continguous ranges of keys
In-Reply-To: <o306ph$vt6$1@blaine.gmane.org>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <1e695bc7-d847-7ac1-e295-393aff699601@gmx.com>
 <o2uvik$qo3$1@blaine.gmane.org>
 <90e54769-94c8-abd3-95a5-4872cb12e86e@gmx.com>
 <o306ph$vt6$1@blaine.gmane.org>
Message-ID: <533a8054-3aa1-2e8e-05b8-6e0f22a26e7a@gmx.com>

On 12/15/2016 11:57 PM, Terry Reedy wrote:
> On 12/15/2016 4:30 PM, Thomas Nyberg wrote:
>> On 12/15/2016 12:48 PM, Terry Reedy wrote:
>>> A sparse array has at least half missing values.  This one has none on
>>> the defined domain, but contiguous dupicates.
>>>
>>
>> I'm sorry for devolving into semantics, but there certainly isn't a
>> single definition of "sparse array" out there. For example, the
>> definition in wikipedia (https://en.wikipedia.org/wiki/Sparse_array)
>> doesn't agree with you:
>
> I think it does ;-).
>
>> "In computer science, a sparse array is an array in which most of the
>> elements have the default value (usually 0 or null)."
>
> ...
 >
>> Personally my usage of sparse arrays in scipy has _always_ had all
>> defined values it's just that the default value was 0. I never deal
>> with "missing" values.
>
> Lucky you.  In statistics and analysis of real, experimental data,
> missing values are often a possibility.  They are always a nuisance,
> sometimes a major one.  When the data are represented by arrays,  rather
> than by some compacted form, some value has to be chosen to represent
> the absence of of data.
>

I thought that the definitions were different because yours said "at 
least half" and the other said "most" in addition to your saying 
"missing values" and the other said "default value". Ignoring the first 
minor point, if by "missing value" you basically mean "default value", 
then yes I agree that the definition is the same. Also I didn't mean 
that I was "lucky" to never have dealt with missing values (strictly 
speaking I have, I just do it rarely), but my point was that I deal with 
sparse matricies/arrays/structures quite often, but I rarely deal with 
"missing values" like nulls/nans/Nones. But that point is moot given 
that you apparently meant the same thing with "missing value" as I did 
with "default value" so I don't think we disagree here.

Taking a step back, the real point of sparse anything is: can you 
represent it in a space/speed efficient manner which is "stable" under 
whatever operations you care about. I.e. if you have a default value of 
0, then you have the property that 0 + x = x and 0 * x = 0 which means 
that sparse matrices with default value 0 are stable under addition and 
multiplication. If you have a default value of nan (or None or null) you 
usually have the convention that nan * x = nan (possibly depends on the 
sign of x) and nan + x = nan which means that a sparse matrix with nans 
as default is also stable under addition and multiplication. If you 
chose a default value of (say) 1 you would run into issues with the 
stability of these operations. It wouldn't mean it's not "sparse", but 
it would mean that the operations you care about might not work as well.

The extension to the thread at and is just that now you have multiple 
default values and the fact that they are not assigned at random, but 
are instead runs of constant values means that you can put a sparse 
structure on this (with a suitable definition of "sparse").

 > Let's devolve to a memory-based language like C. An physical array
 > consisting of sequential memory locations must have *some* bit pattern
 > in every byte and hence in every multibyte block.  If I remember
 > correctly, C malloc initialized bytes to all 0 bits, which is an int
 > value of 0 also.  If there is no meaningful value, there must be a
 > default value that means 'missing'.  0 may mean 'missing'. Null values
 > are by definition non-values.  Python uses None as a Null.  If the
 > example had been populated largely with Nones, I would have called it
 > 'sparse'.

It's not really super pertinent to this discussion, but malloc does not 
guarantee that the values are zeroed. That is guaranteed by calloc though:

	http://man7.org/linux/man-pages/man3/malloc.3.html
	http://man7.org/linux/man-pages/man3/calloc.3p.html

Also the point with a sparse representation is that your default value 
wouldn't exist in memory anywhere and that instead the operations would 
understand that it exists by other factors. For example, a sparse matrix 
with all 0s might* be represented by rows of the form

	i,j,v

where i and j are the row and column indices and v is the value at the 
position. So in this representation we would have as many rows as we 
would non-default values.

*I say might because there are usually more compact forms like this. 
This isn't the internal representation of scipy.sparse.csr_matrix, for 
example.

Regardless, I think I wrote too much to say basically that I don't think 
we're really disagreeing except possibly slightly on perspective.

Cheers,
Thomas


From mail at timgolden.me.uk  Fri Dec 16 13:20:02 2016
From: mail at timgolden.me.uk (Tim Golden)
Date: Fri, 16 Dec 2016 18:20:02 +0000
Subject: Reading python list as a newsgroup (was ...)
In-Reply-To: <122fd551-8bd2-dd24-3bda-41de8e5798ca@vex.net>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com> <o3107q$gm5$1@blaine.gmane.org>
 <122fd551-8bd2-dd24-3bda-41de8e5798ca@vex.net>
Message-ID: <0ef4d743-f339-2d65-0f49-ba9b321a3f08@timgolden.me.uk>

On 16/12/2016 16:12, D'Arcy Cain wrote:
> On 2016-12-16 10:11 AM, Grant Edwards wrote:
>> I didn't notice much spam on c.l.p (but then again, I filter out all
>> posts from google-groups).  The problem on c.l.p that caused me to
>
> Yes, blocking GG was the biggest improvement to my reading this list
> (mailing list in my case).  That and a few judicious PLONKs and it is
> now useful.
>
> Al I need to do now is figure out how to filter out the almost daily
> Windows installer questions.
>

The ever-so-slight irony is that Mailman decided it didn't like this 
post and held it for moderation! (Something to do with the headers; I 
didn't bother to check since I recognised the sender).

TJG


From mbg1708 at planetmail.com  Fri Dec 16 13:39:34 2016
From: mbg1708 at planetmail.com (mbg1708 at planetmail.com)
Date: Fri, 16 Dec 2016 10:39:34 -0800 (PST)
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <ab47d10f-036a-4ad6-98db-8ea0e1e78a5b@googlegroups.com>

On Thursday, 15 December 2016 17:06:39 UTC, Steve D'Aprano  wrote:
> I have some key:value data where the keys often are found in contiguous
> ranges with identical values. For example:
> 
> {1: "foo",
>  2: "foo",
>  3: "foo",
>  # same for keys 4 through 99
>  100: "foo",
>  101: "bar",
>  102: "bar",
>  103: "foobar",
>  104: "bar",
>  105: "foo", 
> }
> ...
> -- 
> Steve

All the answers seem to rely on in-memory solutions.  But isn't the problem a classic data design problem (cf Codd) with two tables.

CREATE TABLE keys   (id    INTEGER NOT NULL PRIMARY KEY,
                     kkey  INTEGER,
                     UNIQUE (kkey) );
                        ## eg id = 999, kkey=101
CREATE TABLE values (id    INTEGER NOT NULL PRIMARY KEY,
                     k_id  INTEGER,
                     value VARCHAR,
                     UNIQUE (k_id, value),
                     FOREIGN KEY (k_id) REFERENCES keys(id));
                        ## eg k_id = 999, value = "bar"

For example, Python/SQLITE can parse the list of key:value pairs.  key is looked up in keys -- and  a keys row is added if the key is new.  The keys id is saved.

k_id--value pair is looked up -- and a row is added if the pair is new.

Some of this can be simplified by relying on SQL to handle non-UNIQUE errors.
This approach may be slower than in-memory processing, but it has almost no database size limits.



From rosuav at gmail.com  Fri Dec 16 16:28:29 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 17 Dec 2016 08:28:29 +1100
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmp_UAQ9sRt7vP_MjzZk_k0dDpCg-Hdfqgcwwz=5v_L7sw@mail.gmail.com>

On Fri, Dec 16, 2016 at 4:06 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> I have about a million or two keys, with a few hundred or perhaps a few
> thousand distinct values. The size of each contiguous group of keys with
> the same value can vary from 1 to perhaps a hundred or so.

Going right back to the beginning here: I think that "a million or
two" is a perfectly doable figure for a straight-forward list or dict.
You get immediately-available lookups in a straight-forward way, at
the cost of maybe 16MB of memory (if you use the same strings for the
values, the cost is just the pointers).

ChrisA


From johnpote at jptechnical.co.uk  Fri Dec 16 18:15:11 2016
From: johnpote at jptechnical.co.uk (John Pote)
Date: Fri, 16 Dec 2016 23:15:11 +0000
Subject: Mapping with continguous ranges of keys
In-Reply-To: <5853f9e0$0$1610$c3e8da3$5496439d@news.astraweb.com>
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <5853f9e0$0$1610$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <0bf12f70-55b6-860b-cd08-5d5b0bece69e@jptechnical.co.uk>


On 16/12/2016 14:27, Steve D'Aprano wrote:
> (2) The clever solution: use a pair of lists, one holding the starting value
> of each group of keys, and the other holding the common values. This saves
> a lot of memory, but is slower.
>
> A concrete example might help. Suppose I have 15 keys in five groups:
>
> D = {0: 10,
>       1: 20, 2: 20,
>       3: 30, 4: 30, 5: 30,
>       6: 40, 7: 40, 8: 40, 9: 40,
>       10: 50, 11: 50, 12: 50, 13: 50, 14: 50}
>
> (Remember, in reality I could have as many as a million or two keys. This is
> just a simple toy example.)
>
> Instead of using a dict, I also set up a pair of lists:
>
> L = [0, 1, 3, 6, 10, 15]  # starting value of each group
> V = [10, 20, 30, 40, 50]  # value associated with each group
I misread the problem (skipped the comment about keys 4 - 99) and 
assumed there might be gaps between the contiguous blocks so thought of 
the list structure
[  ( (firstKeyN, lastKeyN), "value" ),
     ...
]
At the cost of more memory keeping first and last keys numbers in tuples 
in the L list would mean there is only one L lookup at the expence of 
two additional tuple lookups. Are small tuple lookups quicker than 1 
large list lookup? If not stick with the simple L and V you show above.

I've never used any form of tree structure, perhaps someone else could 
comment of the performance of balanced trees as compared to simple 
lists. Would the insertion cost be too high in keeping the tree balanced?

As to speeding up access with only L and V lists the binary search must 
be optimal unless specialised knowledge about the distribution of the 
size of the contiguous groups is made use of. But you say there is no 
pattern to the group sizes.

Other thought is to have a smaller pre-index list, search this to find 
the range of L indexes the key is in. If the pre-index list had a 1000 
entries then each entry would cover 1/1000 of the L list which narrows 
the binary search space in L considerably. The cost of something like 
this is keeping the pre-index list up to date when new keys are added 
and extra time to code and test it.
preList struct [ (firstKeyN, lastKeyN), (firstLindex, lastLindex),
                         ...
                       ]

Reminds me of Jackson's first two rules on optimisation, 1 - don't do 
it, 2 - don't do it yet
Thanks for an interesting problem.
> Note that the first list has one extra item, namely the number one past the
> final group.
>
> I can do a key look-up using either of these:
>
>      D[key]
>
>      V[bisect.bisect_right(L, i) - 1]
>
>
> I tested the memory consumption and speed of these two solutions with
> (approximately) one million keys. I found:
>
> - the dict solution uses a lot more memory, about 24 bytes per key, compared
> to the pair of lists solution, which is about 0.3 bytes per key;
>
> - but on my computer, dict lookups are about 3-4 times faster.
>
>
> Any suggestions for improving the speed of the binary search version, or the
> memory consumption of the dict?
>
> By the way: the particular pattern of groups in the sample code (groups of
> size 1, 2, 3, ... up to 50, then repeating from 1, 2, 3, ... again) is just
> demo. In my real data, the sizes of the groups are all over the place, in
> an unpredictable pattern.
>
>
>
> Thanks in advance.
>
>



From sohcahtoa82 at gmail.com  Fri Dec 16 21:20:00 2016
From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com)
Date: Fri, 16 Dec 2016 18:20:00 -0800 (PST)
Subject: Cache memory and its effect on list searching
Message-ID: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>

Alternatively...why you should definitely use binary searches:

Python 3.5.2+ (default, Aug 30 2016, 19:08:42) 
[GCC 6.2.0 20160822] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> import timeit
>>> hashes =  [hashlib.md5(bytes(str(i), "utf-8")).hexdigest() for i in range(10000000)]
>>> sortedhashes = sorted(hashes)
>>> timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10)
1.9478233020054176
>>> timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10)
18.001392804995703

I thought this was curious behavior.  I created a list of random-looking strings, then made a sorted copy.  I then found that using "in" to see if a string exists in the sorted list took over 9 times as long!

At first, I thought since both lists are the same size, and the 'in' test is a linear search, shouldn't they take the same amount of time?  Even if there was some trickery with branch prediction happening, that would have benefited the sorted list.  Then I remembered how lists work in Python.  The original list is going to be mostly contiguous in memory, making the memory cache quite effective.  When I create the sorted copy, I'm creating a list of references to strings that are all over the place in memory, causing tons of cache misses.

Of course, the best solution was to implement a binary search.  That turned the membership check into a 300-700 microsecond operation.


From rosuav at gmail.com  Fri Dec 16 21:27:01 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 17 Dec 2016 13:27:01 +1100
Subject: Cache memory and its effect on list searching
In-Reply-To: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
References: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
Message-ID: <CAPTjJmphYxajjTeF3AP7rYco+bC1fs_Doc_Wp=4nNVYjQxaL-A@mail.gmail.com>

On Sat, Dec 17, 2016 at 1:20 PM,  <sohcahtoa82 at gmail.com> wrote:
> I thought this was curious behavior.  I created a list of random-looking strings, then made a sorted copy.  I then found that using "in" to see if a string exists in the sorted list took over 9 times as long!
>

My numbers replicate yours (though my computer's faster). But my
conclusion is different:

Python 3.7.0a0 (default:ecd218c41cd4, Dec 16 2016, 03:08:47)
[GCC 6.2.0 20161027] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> import timeit
>>> hashes =  [hashlib.md5(bytes(str(i), "utf-8")).hexdigest() for i in range(10000000)]
>>> sortedhashes = sorted(hashes)
>>> timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10)
0.8167107938788831
>>> timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10)
5.029693723190576
>>> timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10)
0.855183657258749
>>> timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10)
5.0585526106879115
>>> sethashes = set(hashes)
>>> timeit.timeit('"x" in hashes', globals={'hashes': sethashes}, number=10)
6.13601878285408e-06

You want containment checks? Use a set :)

ChrisA


From darcy at vex.net  Fri Dec 16 22:52:52 2016
From: darcy at vex.net (D'Arcy Cain)
Date: Fri, 16 Dec 2016 22:52:52 -0500
Subject: Reading python list as a newsgroup (was ...)
In-Reply-To: <sa495cpo18tb6d35cq5urdbfhe3rf5avno@4ax.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com> <o3107q$gm5$1@blaine.gmane.org>
 <sa495cpo18tb6d35cq5urdbfhe3rf5avno@4ax.com>
Message-ID: <97e85e9a-f6a9-dd85-c4c5-e825d1c4de91@vex.net>

On 2016-12-16 08:16 PM, Dennis Lee Bieber wrote:
> 	Unfortunately, my client can only "pre filter" on subject and author; I
> could kill all @gmail.*, but could not focus on just Google Groups
> submissions...

I don't know what client you use but perhaps you can adapt this procmail 
recipe.

:0 Hir
* ^List-Id:.*python-list.python.org
* ^From:.*@gmail.com
* ^Newsgroups:.*
/dev/null


-- 
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:darcy at Vex.Net
VoIP: sip:darcy at Vex.Net


From spluque at gmail.com  Fri Dec 16 22:56:12 2016
From: spluque at gmail.com (Seb)
Date: Fri, 16 Dec 2016 21:56:12 -0600
Subject: building numpy arrays with regular structure
Message-ID: <87twa3p5sz.fsf@otaria.sebmel.org>

Hello,

Is there an easier way to write a numpy array with a regular structure?
For example, an array with [0, 1] along the diagnal of one of the array
dimensions, and zero elsewhere:

zz = np.array([[[0, 1], [0, 0], [0, 0]],
               [[0, 0], [0, 1], [0, 0]],
               [[0, 0], [0, 0], [0, 1]]])

This one is not so big, but if it were, there must be a way to code this
properly.

Thanks,

-- 
Seb



From vek.m1234 at gmail.com  Fri Dec 16 23:42:28 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sat, 17 Dec 2016 10:12:28 +0530
Subject: Is there a way to insert hooks into a native dictionary type to
 see when a query arrives and what's looked up?
References: <o2qno3$oqp$1@dont-email.me>
 <58511326$0$2775$c3e8da3$76491128@news.astraweb.com>
Message-ID: <o32fkv$akm$1@dont-email.me>

Steven D'Aprano wrote:

> On Wednesday 14 December 2016 17:11, Veek M wrote:
> 
>> I know that with user classes one can define getattr, setattr to
>> handle dictionary lookup. Is there a way to hook into the native
>> dict() type and see in real time what's being queried.
> 
> Not easily, and maybe not at all.
> 
> There are two obvious ways to do this:
> 
> (1) monkey-patch the object's __dict__, and the class __dict__.
> 
> Unfortunately, Python doesn't support monkey-patching built-ins.
> 
> https://en.wikipedia.org/wiki/Monkey_patch
> 
> Or perhaps I should say, *fortunately* Python doesn't support it.
> 
> http://www.virtuouscode.com/2008/02/23/why-monkeypatching-is-destroying-ruby/
> 
> (2) Alternatively, you could make a dict subclass, and replace the
> class and instance __dict__ with your own.
> 
> Unfortunately, you cannot replace the __dict__ of a class:
> 
> py> class X:  # the class you want to hook into
> ...     pass
> ...
> py> class MyDict(dict):  # my custom dict
> ...     def __getitem__(self, key):
> ...             print(key)
> ...             return super().__getitem__(key)
> ...
> py> d = MyDict()
> py> d.update(X.__dict__)
> py> X.__dict__ = d
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: attribute '__dict__' of 'type' objects is not writable
> 
> 
> You can replace the instance dict, but Python won't call your
> __getitem__ method:
> 
> py> instance = X()
> py> instance.__dict__ = MyDict()
> py> instance.a = 999
> py> instance.a
> 999
> 
> So the short answer is, No.
> 
> You might be able to create a completely new metaclass that supports
> this, but it would be a lot of work, and I'm not even sure that it
> would be successful.
> 
> 
> 
>> I wanted to check if when one does:
>> 
>> x.sin()
>> 
>> if the x.__dict__ was queried or if the Foo.__dict__ was queried..
> 
> The easiest way to do that is something like this:
> 
> 
> py> class Test:
> ...     def sin(self):
> ...             return 999
> ...
> py> x = Test()
> py> x.sin
> <bound method Test.sin of <__main__.Test object at 0xb6fc3a4c>>
> py> x.sin()
> 999
> py> x.sin = "surprise!"
> py> x.sin
> 'surprise!'
> 
> 
> 
> So now you know: an instance attribute will shadow the class
> attribute.
> 
> (Actually, that's not *completely* true. It depends on whether x.sin
> is a descriptor or not, and if so, what kind of descriptor.)
> 
> 

heh
If it walks like a duck and talks like a duck, it?s a duck, right? So if 
this duck is not giving you the noise that you want, you?ve got to just 
punch that duck until it returns what you expect. -Patrick Ewing on 
Monkey/Duck patching in RailsConf 2007


From sohcahtoa82 at gmail.com  Fri Dec 16 23:44:12 2016
From: sohcahtoa82 at gmail.com (sohcahtoa82 at gmail.com)
Date: Fri, 16 Dec 2016 20:44:12 -0800 (PST)
Subject: Cache memory and its effect on list searching
In-Reply-To: <mailman.131.1481941629.2322.python-list@python.org>
References: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
 <CAPTjJmphYxajjTeF3AP7rYco+bC1fs_Doc_Wp=4nNVYjQxaL-A@mail.gmail.com>
 <mailman.131.1481941629.2322.python-list@python.org>
Message-ID: <64c48b00-550a-4867-b60a-21689f22f770@googlegroups.com>

On Friday, December 16, 2016 at 6:27:24 PM UTC-8, Chris Angelico wrote:
> On Sat, Dec 17, 2016 at 1:20 PM,  <sohcahtoa82 at gmail.com> wrote:
> > I thought this was curious behavior.  I created a list of random-looking strings, then made a sorted copy.  I then found that using "in" to see if a string exists in the sorted list took over 9 times as long!
> >
> 
> My numbers replicate yours (though my computer's faster). But my
> conclusion is different:
> 
> Python 3.7.0a0 (default:ecd218c41cd4, Dec 16 2016, 03:08:47)
> [GCC 6.2.0 20161027] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import hashlib
> >>> import timeit
> >>> hashes =  [hashlib.md5(bytes(str(i), "utf-8")).hexdigest() for i in range(10000000)]
> >>> sortedhashes = sorted(hashes)
> >>> timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10)
> 0.8167107938788831
> >>> timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10)
> 5.029693723190576
> >>> timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10)
> 0.855183657258749
> >>> timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10)
> 5.0585526106879115
> >>> sethashes = set(hashes)
> >>> timeit.timeit('"x" in hashes', globals={'hashes': sethashes}, number=10)
> 6.13601878285408e-06
> 
> You want containment checks? Use a set :)
> 
> ChrisA

Ah, I forgot about the optimizations of a set.  The only time I ever find myself using set is when I want to remove all the duplicates in a list.  I convert it to a set and then back.

For fun, I ran an experiment:

### BEGIN listsearch.py
import hashlib 
import timeit
import sys

from bisect import bisect_left

def bisect_search(a, x, lo=0, hi=None):   # can't use a to specify default for hi
    # From http://stackoverflow.com/questions/212358/binary-search-bisection-in-python
    hi = hi if hi is not None else len(a) # hi defaults to len(a)
    pos = bisect_left(a,x,lo,hi)          # find insertion position
    return (pos if pos != hi and a[pos] == x else -1) # don't walk off the end

def bin_search(haystack, needle):
    start = 0
    end = len(haystack) - 1
    while True:
        if start > end:
            return False
        middle = (start + end) // 2
        if needle < haystack[middle]:
            end = middle - 1
            continue
        elif needle > haystack[middle]:
            start = middle + 1
            continue
        elif needle == haystack[middle]:
            return True

print('Python version: ', sys.version_info)
print('building hashes...')
hashes = [hashlib.md5(bytes(str(i), "utf-8")).hexdigest() for i in range(10000000)]
print('sorting...')
sortedhashes = sorted(hashes)
print('creating set...')
sethashes = set(hashes)
print('Unsorted list:', timeit.timeit('"x" in hashes', globals={'hashes': hashes}, number=10))
print('Sorted:', timeit.timeit('"x" in hashes', globals={'hashes': sortedhashes}, number=10))
print('set:', timeit.timeit('"x" in hashes', globals={'hashes': sethashes}, number=10))
print('binary search:',  timeit.timeit('binsearch(hashes, "x")', globals={'hashes': sortedhashes, 'binsearch': bin_search}, number=10))
print('binary search with bisect:',  timeit.timeit('binsearch(hashes, "x")', globals={'hashes': sortedhashes, 'binsearch': bisect_search}, number=10))
### END listsearch.py

> python3 listsearch.py
Python version:  sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
building hashes...
sorting...
creating set...
Unsorted list: 1.7384763684627569
Sorted: 9.248799958145042
set: 1.4614161294446149e-06
binary search: 0.00010902164328108199
binary search with bisect: 1.7829276782066472e-05

Yup.  set is definitely the way to go!


From rosuav at gmail.com  Fri Dec 16 23:51:17 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 17 Dec 2016 15:51:17 +1100
Subject: Is there a way to insert hooks into a native dictionary type to
 see when a query arrives and what's looked up?
In-Reply-To: <o32fkv$akm$1@dont-email.me>
References: <o2qno3$oqp$1@dont-email.me>
 <58511326$0$2775$c3e8da3$76491128@news.astraweb.com>
 <o32fkv$akm$1@dont-email.me>
Message-ID: <CAPTjJmrDcKeLqDY2LFsJ3LmNOUGLF5v8KXVY9h4TeVLZpKO5nw@mail.gmail.com>

On Sat, Dec 17, 2016 at 3:42 PM, Veek M <vek.m1234 at gmail.com> wrote:
> If it walks like a duck and talks like a duck, it?s a duck, right? So if
> this duck is not giving you the noise that you want, you?ve got to just
> punch that duck until it returns what you expect. -Patrick Ewing on
> Monkey/Duck patching in RailsConf 2007

https://www.youtube.com/watch?v=PFn_agxmatg

ChrisA


From rosuav at gmail.com  Fri Dec 16 23:55:01 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 17 Dec 2016 15:55:01 +1100
Subject: Cache memory and its effect on list searching
In-Reply-To: <64c48b00-550a-4867-b60a-21689f22f770@googlegroups.com>
References: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
 <CAPTjJmphYxajjTeF3AP7rYco+bC1fs_Doc_Wp=4nNVYjQxaL-A@mail.gmail.com>
 <mailman.131.1481941629.2322.python-list@python.org>
 <64c48b00-550a-4867-b60a-21689f22f770@googlegroups.com>
Message-ID: <CAPTjJmpkSROAAbT8D9kr6VTne_6+YYh8=_hoYQkX41kJPOYA-g@mail.gmail.com>

On Sat, Dec 17, 2016 at 3:44 PM,  <sohcahtoa82 at gmail.com> wrote:
>> python3 listsearch.py
> Python version:  sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
> building hashes...
> sorting...
> creating set...
> Unsorted list: 1.7384763684627569
> Sorted: 9.248799958145042
> set: 1.4614161294446149e-06
> binary search: 0.00010902164328108199
> binary search with bisect: 1.7829276782066472e-05
>
> Yup.  set is definitely the way to go!

More than that: the lists are searched in linear time, the binary
seach runs in logarithmic time, but the set lookup is constant time.
Doesn't matter how big your set is, you can test for membership with
one hash lookup.

ChrisA


From nad at python.org  Sat Dec 17 00:01:00 2016
From: nad at python.org (Ned Deily)
Date: Sat, 17 Dec 2016 00:01:00 -0500
Subject: [RELEASE] Python 3.6.0rc2 is now available
Message-ID: <281D809C-E94E-41BB-954D-D092A4CBC03B@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I would like to announce the availability of Python 3.6.0rc2. 3.6.0rc2
is the second release candidate for Python 3.6, the next major release of
Python.

Code for 3.6.0 is now frozen.  3.6.0rc2 is the same code base as the first
release candidate, 3.6.0rc1, with the addition of fixes for a couple of
critical problems and with some documentation additions and updates.
Assuming no further release critical problems are found prior to the 3.6.0
final release date, now planned for 2016-12-23, the 3.6.0 final release
will be the same code base as this 3.6.0rc2.  Maintenance releases for the
3.6 series will follow at regular intervals starting in the first quarter
of 2017.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8
* PEP 529 - Change Windows filesystem encoding to UTF-8
* PEP 530 - Asynchronous Comprehensions

Please see "What?s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0rc2 here:

https://www.python.org/downloads/release/python-360rc2/

Note that 3.6.0rc2 is still a preview release and thus its use is not
recommended for production environments.

More information about the release schedule can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []



From steve+python at pearwood.info  Sat Dec 17 01:12:03 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 17 Dec 2016 17:12:03 +1100
Subject: Cache memory and its effect on list searching
References: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
 <CAPTjJmphYxajjTeF3AP7rYco+bC1fs_Doc_Wp=4nNVYjQxaL-A@mail.gmail.com>
 <mailman.131.1481941629.2322.python-list@python.org>
 <64c48b00-550a-4867-b60a-21689f22f770@googlegroups.com>
 <CAPTjJmpkSROAAbT8D9kr6VTne_6+YYh8=_hoYQkX41kJPOYA-g@mail.gmail.com>
 <mailman.135.1481950504.2322.python-list@python.org>
Message-ID: <5854d734$0$1606$c3e8da3$5496439d@news.astraweb.com>

On Sat, 17 Dec 2016 03:55 pm, Chris Angelico wrote:

> More than that: the lists are searched in linear time, the binary
> seach runs in logarithmic time, but the set lookup is constant time.
> Doesn't matter how big your set is, you can test for membership with
> one hash lookup.

To be pedantic: it is constant time on average.

Best case is one hash and one equality test. Worst case is if all the set
elements have colliding hashes, in which case it degenerates to a linear
search.

There is a class of denial of service attacks where the attacker can specify
the keys in a dict in such a way that lookups collide, and can push new
colliding items into the dictionary (or set) faster than Python can perform
the lookups. That's why Python now has hash randomisation:

http://bugs.python.org/issue13703
https://www.python.org/dev/peps/pep-0456/


But outside of contrived or hostile examples, where elements of the set are
designed to collide, typically you would expect hash collisions to be rare,
and long chains of colliding elements even rarer.  For random elements,
most will require only a single hash and a single equality test, a small
number might require two tests, three would be even rarer, and so forth.

So strictly speaking, and I realise I'm being exceedingly pedantic here, a
*sufficiently large* dict or set MUST have colliding elements. How large
is "sufficiently large"? Its at least 2**31, more than two billion, so
while you are right that *in practice* set/dict lookups require only a
single hash + equality, in principle (and sometimes in practice too)
collisions can be significant.

Nevertheless, I mention this only for completeness. In practice, you almost
never need to care about hash collisions except for the most pathological
cases.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sat Dec 17 01:34:03 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 17 Dec 2016 17:34:03 +1100
Subject: Cache memory and its effect on list searching
In-Reply-To: <5854d734$0$1606$c3e8da3$5496439d@news.astraweb.com>
References: <447773dd-df7f-4251-8e1d-43260539ec8b@googlegroups.com>
 <CAPTjJmphYxajjTeF3AP7rYco+bC1fs_Doc_Wp=4nNVYjQxaL-A@mail.gmail.com>
 <mailman.131.1481941629.2322.python-list@python.org>
 <64c48b00-550a-4867-b60a-21689f22f770@googlegroups.com>
 <CAPTjJmpkSROAAbT8D9kr6VTne_6+YYh8=_hoYQkX41kJPOYA-g@mail.gmail.com>
 <mailman.135.1481950504.2322.python-list@python.org>
 <5854d734$0$1606$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpxsdRdiK5wEvDQp7v6Bb6AsNMhLy9+SoBKmT=8tpXRzw@mail.gmail.com>

On Sat, Dec 17, 2016 at 5:12 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Sat, 17 Dec 2016 03:55 pm, Chris Angelico wrote:
>
>> More than that: the lists are searched in linear time, the binary
>> seach runs in logarithmic time, but the set lookup is constant time.
>> Doesn't matter how big your set is, you can test for membership with
>> one hash lookup.
>
> To be pedantic: it is constant time on average.

True. And technically, you could say that a linear or binary search
has a best-case of constant time (if you find it right at the
beginning or middle of the list). I was talking about the average.

> But outside of contrived or hostile examples, where elements of the set are
> designed to collide, typically you would expect hash collisions to be rare,
> and long chains of colliding elements even rarer.  For random elements,
> most will require only a single hash and a single equality test, a small
> number might require two tests, three would be even rarer, and so forth.

Yep. Normal case, it's going to be one or two tests - and that doesn't
depend on the length of the list. The degenerate case does, but the
normal case doesn't.

> So strictly speaking, and I realise I'm being exceedingly pedantic here, a
> *sufficiently large* dict or set MUST have colliding elements. How large
> is "sufficiently large"? Its at least 2**31, more than two billion, so
> while you are right that *in practice* set/dict lookups require only a
> single hash + equality, in principle (and sometimes in practice too)
> collisions can be significant.

That assumes the hash has a finite size. If you have enough memory to
store that many unique objects, you can probably afford to use a
hashing algorithm that allows enough room. However, the chances of not
having collisions depend on the capacity. And the chances of having no
collisions at all are pretty low... as a rule of thumb, I estimate on
a 50-50 chance of a collision when capacity is the square of usage. So
if you have a thousand entries but capacity for a million, you have a
50% chance of having at least one collision. (The numbers aren't quite
right, but it's a good rule of thumb.)

> Nevertheless, I mention this only for completeness. In practice, you almost
> never need to care about hash collisions except for the most pathological
> cases.

Indeed. That's why hash tables are so prevalent, despite their
appalling worst-case.

ChrisA


From __peter__ at web.de  Sat Dec 17 03:54:07 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 17 Dec 2016 09:54:07 +0100
Subject: building numpy arrays with regular structure
References: <87twa3p5sz.fsf@otaria.sebmel.org>
Message-ID: <o32ufb$ggb$1@blaine.gmane.org>

Seb wrote:

> Is there an easier way to write a numpy array with a regular structure?
> For example, an array with [0, 1] along the diagnal of one of the array
> dimensions, and zero elsewhere:
> 
> zz = np.array([[[0, 1], [0, 0], [0, 0]],
>                [[0, 0], [0, 1], [0, 0]],
>                [[0, 0], [0, 0], [0, 1]]])
> 
> This one is not so big, but if it were, there must be a way to code this
> properly.

Searching for "numpy assigning diagonal values" gave

https://docs.scipy.org/doc/numpy/reference/generated/numpy.fill_diagonal.html

as the first hit. So

>>> a = numpy.zeros((3,3,2), dtype=int)
>>> a
array([[[0, 0],
        [0, 0],
        [0, 0]],

       [[0, 0],
        [0, 0],
        [0, 0]],

       [[0, 0],
        [0, 0],
        [0, 0]]])
>>> numpy.fill_diagonal(a[:,:,1], 1)
>>> a
array([[[0, 1],
        [0, 0],
        [0, 0]],

       [[0, 0],
        [0, 1],
        [0, 0]],

       [[0, 0],
        [0, 0],
        [0, 1]]])




From __peter__ at web.de  Sat Dec 17 04:31:40 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 17 Dec 2016 10:31:40 +0100
Subject: Mapping with continguous ranges of keys
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <5853f9e0$0$1610$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o330lo$6cl$1@blaine.gmane.org>

Steve D'Aprano wrote:

> I have experimented with two solutions, and hope somebody might be able to
> suggest some improvements. Attached is the test code I ran, suggestions
> for improving the performance will be appreciated.

If there was an attachment to this text -- that didn't make it to the 
mailing list or the news group.

> I decided on these two data structures:
> 
> (1) The naive solution: don't worry about duplicate values, or the fact
> that keys come in contiguous groups, and just store each individual key
> and value in a dict; this is faster but uses more memory.
> 
> (2) The clever solution: use a pair of lists, one holding the starting
> value of each group of keys, and the other holding the common values. This
> saves a lot of memory, but is slower.
> 
> A concrete example might help. Suppose I have 15 keys in five groups:
> 
> D = {0: 10,
>      1: 20, 2: 20,
>      3: 30, 4: 30, 5: 30,
>      6: 40, 7: 40, 8: 40, 9: 40,
>      10: 50, 11: 50, 12: 50, 13: 50, 14: 50}
> 
> (Remember, in reality I could have as many as a million or two keys. This
> is just a simple toy example.)
> 
> Instead of using a dict, I also set up a pair of lists:
> 
> L = [0, 1, 3, 6, 10, 15]  # starting value of each group
> V = [10, 20, 30, 40, 50]  # value associated with each group
> 
> Note that the first list has one extra item, namely the number one past
> the final group.
> 
> I can do a key look-up using either of these:
> 
>     D[key]
> 
>     V[bisect.bisect_right(L, i) - 1]
> 
> 
> I tested the memory consumption and speed of these two solutions with
> (approximately) one million keys. I found:
> 
> - the dict solution uses a lot more memory, about 24 bytes per key,
> compared to the pair of lists solution, which is about 0.3 bytes per key;
> 
> - but on my computer, dict lookups are about 3-4 times faster.

Only three to four times? You basically get that from a no-op function call:

$ python3 -m timeit -s 'd = {1: 2}; k = 1' 'd[k]'
10000000 loops, best of 3: 0.0935 usec per loop
$ python3 -m timeit -s 'd = {1: 2}; k = 1' 'd[int(k)]'
1000000 loops, best of 3: 0.304 usec per loop

Even adding a dummy value to V to go from

>     V[bisect.bisect_right(L, i) - 1]

to

V[bisect.bisect_right(L, i)]

might be visible in your benchmark.

> Any suggestions for improving the speed of the binary search version, or
> the memory consumption of the dict?

Depending on the usage pattern you might try bisect combined with a LRU 
cache. 

If runs of four or more nearby keys are common you can remember the current 
span:

# untested, watch out for off-by-one errors ;)
start = stop = 0
for key in data:
    if start <= key < stop:
        pass # reuse value
    else:
        index = bisect.bisect_right(L, key)
        start, stop = L[index: index + 2]
        value = V[index - 1]

    # use value

(You might also fill V with (value, start, stop) tuples))

> By the way: the particular pattern of groups in the sample code (groups of
> size 1, 2, 3, ... up to 50, then repeating from 1, 2, 3, ... again) is
> just demo. In my real data, the sizes of the groups are all over the
> place, in an unpredictable pattern.
> 
> 
> 
> Thanks in advance.
> 
> 




From smith at smith.it  Sat Dec 17 05:36:29 2016
From: smith at smith.it (Smith)
Date: Sat, 17 Dec 2016 11:36:29 +0100
Subject: To improve a script
Message-ID: <o334fc$1r4v$1@gioia.aioe.org>

Hello guys,
can someone help me to improve this script?

https://github.com/githubdavide/mitm/blob/master/mitm.py

Thank you in advance

Cheers




From steve+python at pearwood.info  Sat Dec 17 06:24:25 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 17 Dec 2016 22:24:25 +1100
Subject: Mapping with continguous ranges of keys
References: <5852cd93$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <5853f9e0$0$1610$c3e8da3$5496439d@news.astraweb.com>
 <o330lo$6cl$1@blaine.gmane.org>
 <mailman.140.1481967113.2322.python-list@python.org>
Message-ID: <5855206b$0$1616$c3e8da3$5496439d@news.astraweb.com>

On Sat, 17 Dec 2016 08:31 pm, Peter Otten wrote:

> Steve D'Aprano wrote:
> 
>> I have experimented with two solutions, and hope somebody might be able
>> to suggest some improvements. Attached is the test code I ran,
>> suggestions for improving the performance will be appreciated.
> 
> If there was an attachment to this text -- that didn't make it to the
> mailing list or the news group.

I see it on comp.lang.python but not the mailing list or gmane. That's
annoying because its a *text* attachment and should be allowed. I'll attach
it again, inline this time, at the end of this post.


[...]
>> I tested the memory consumption and speed of these two solutions with
>> (approximately) one million keys. I found:
>> 
>> - the dict solution uses a lot more memory, about 24 bytes per key,
>> compared to the pair of lists solution, which is about 0.3 bytes per key;
>> 
>> - but on my computer, dict lookups are about 3-4 times faster.
> 
> Only three to four times? You basically get that from a no-op function
> call:
[...]
> Even adding a dummy value to V to go from
> 
>>     V[bisect.bisect_right(L, i) - 1]
> 
> to
> 
> V[bisect.bisect_right(L, i)]
> 
> might be visible in your benchmark.

Good thinking! I'll try that.

And... it doesn't appear to make any real difference.


>> Any suggestions for improving the speed of the binary search version, or
>> the memory consumption of the dict?
> 
> Depending on the usage pattern you might try bisect combined with a LRU
> cache.

Adding a cache will probably eat up the memory savings, but I may play
around with that.

And here's the code, this time inline. I've added the dummy value to the
values list V as suggested.

# --- cut ---

import bisect
import random
import sys

from timeit import default_timer as timer

def make_keys():
    """Yield (start, end) values suitable for passing to range().

    Distance between start and end increase from 1 to 51, then
    repeat, for a total of 800*25*51 = 1020000 values all together.
    """
    n = 1
    for j in range(800):
        for i in range(50):
            yield (n, n+i+1)
            n += i+1

def populate():
    D = {}
    L = []
    V = [None]
    value = 1
    last_b = 1
    for a, b in make_keys():
        assert a == last_b
        for i in range(a, b):
            D[i] = value
        L.append(a)
        V.append(value)
        last_b = b
    L.append(last_b)
    return (D, L, V)


class Stopwatch:
    """Context manager for timing long running chunks of code."""

    def __enter__(self):
        self._start = timer()
        return self

    def __exit__(self, *args):
        elapsed = timer() - self._start
        del self._start
        if elapsed < 0.01:
            print("Elapsed time is very small, consider using timeit
instead.")
        print('time taken: %f seconds' % elapsed)



D, L, V = populate()
assert len(D) == 800*25*51
assert len(L) == len(V)
print("size of dict", sys.getsizeof(D))
print("size of two lists", sys.getsizeof(L) + sys.getsizeof(V))


# Confirm that values are the same whether using dict lookup
# or binary search.
for i in range(1, 800*25*51 + 1):
    index = bisect.bisect_right(L, i)
    assert D[i] == V[index]


# Simulate a large number of lookups in random order.
nums = list(range(1, 800*25*51 + 1))
random.shuffle(nums)

with Stopwatch():
    for i in nums:
        x = D[i]

with Stopwatch():
    for i in nums:
        x = V[bisect.bisect_right(L, i)]

# --- cut ---



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ben+python at benfinney.id.au  Sat Dec 17 07:42:48 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 17 Dec 2016 23:42:48 +1100
Subject: To improve a script
References: <o334fc$1r4v$1@gioia.aioe.org>
Message-ID: <85h962zpyv.fsf@benfinney.id.au>

Smith <smith at smith.it> writes:

> Hello guys,
> can someone help me to improve this script?

Yes. Remove it from every repository; the world will be improved by
removing such hostility from the internet.

-- 
 \        ?When you go in for a job interview, I think a good thing to |
  `\                  ask is if they ever press charges.? ?Jack Handey |
_o__)                                                                  |
Ben Finney



From grant.b.edwards at gmail.com  Sat Dec 17 12:21:34 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sat, 17 Dec 2016 17:21:34 +0000 (UTC)
Subject: Reading python list as a newsgroup (was ...)
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com>
 <o3107q$gm5$1@blaine.gmane.org>
 <sa495cpo18tb6d35cq5urdbfhe3rf5avno@4ax.com>
Message-ID: <o33s6u$15t$1@blaine.gmane.org>

On 2016-12-17, Dennis Lee Bieber <wlfraed at ix.netcom.com> wrote:
> On Fri, 16 Dec 2016 15:11:54 +0000 (UTC), Grant Edwards
><grant.b.edwards at gmail.com> declaimed the following:
>
>>I didn't notice much spam on c.l.p (but then again, I filter out all
>>posts from google-groups).  The problem on c.l.p that caused me to
>>switch to gmane's NNTP server was the number of broken references.
>>They both have references that break in various scenarios, but my
>>tests indicated that it was worse on c.l.p.  [Yes, I actually wrote a
>>Python program that used an NNTP client library to check all the
>>reference values in a large sample of posts from both sources.]
>
> 	Unfortunately, my client can only "pre filter" on subject and author; I
> could kill all @gmail.*, but could not focus on just Google Groups
> submissions...
>
> 	Probably time for me to spend the $19 to upgrade from Agent 6.0

Or use the One True Newsreader: SLRN.

;)

--
Grant




From grant.b.edwards at gmail.com  Sat Dec 17 12:23:01 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sat, 17 Dec 2016 17:23:01 +0000 (UTC)
Subject: Reading python list as a newsgroup (was ...)
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <532ecc1d-8245-8bd8-d868-22f4ff9a9fde@udel.edu>
 <ojh65cdooa1i2ivr63fg3p3flbalphp36a@4ax.com>
 <o3107q$gm5$1@blaine.gmane.org>
 <sa495cpo18tb6d35cq5urdbfhe3rf5avno@4ax.com>
 <97e85e9a-f6a9-dd85-c4c5-e825d1c4de91@vex.net>
Message-ID: <o33s9l$15t$2@blaine.gmane.org>

On 2016-12-17, D'Arcy Cain <darcy at vex.net> wrote:
> On 2016-12-16 08:16 PM, Dennis Lee Bieber wrote:
>> 	Unfortunately, my client can only "pre filter" on subject and author; I
>> could kill all @gmail.*, but could not focus on just Google Groups
>> submissions...
>
> I don't know what client you use but perhaps you can adapt this procmail 
> recipe.
>
>:0 Hir
> * ^List-Id:.*python-list.python.org
> * ^From:.*@gmail.com
> * ^Newsgroups:.*
> /dev/null

That's not what he wants to do.  He wants to filter out posts made via
Google Groups, not posts sent from people who use gmail addresses.

Here's the rule for slrn:

Score:: =-9999
 Message-ID: .*googlegroups.com
 




From miaojpm at gmail.com  Sat Dec 17 14:10:22 2016
From: miaojpm at gmail.com (John)
Date: Sat, 17 Dec 2016 11:10:22 -0800 (PST)
Subject: python list index - an easy question
Message-ID: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>

Hi, 

   I am new to Python, and I believe it's an easy question. I know R and Matlab.

************
>>> x=[1,2,3,4,5,6,7]
>>> x[0]
1
>>> x[1:5]
[2, 3, 4, 5]
*************

    My question is: what does x[1:5] mean? By Python's convention, the first element of a list is indexed as "0". Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? If I am right, it should print [2,3,4,5,6]. Why does it print only [2,3,4,5]?

   Thanks!!

John


From robertvstepp at gmail.com  Sat Dec 17 14:24:17 2016
From: robertvstepp at gmail.com (boB Stepp)
Date: Sat, 17 Dec 2016 13:24:17 -0600
Subject: python list index - an easy question
In-Reply-To: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <CANDiX9J9GQTr_WXeW6OsnkXm57SGrK6VvamEp3tLm8AvMLKd5Q@mail.gmail.com>

On Sat, Dec 17, 2016 at 1:10 PM, John <miaojpm at gmail.com> wrote:
>
> Hi,
>
>    I am new to Python, and I believe it's an easy question. I know R and Matlab.
>
> ************
> >>> x=[1,2,3,4,5,6,7]
> >>> x[0]
> 1
> >>> x[1:5]
> [2, 3, 4, 5]
> *************
>
>     My question is: what does x[1:5] mean? By Python's convention, the first element of a list is indexed as "0". Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? If I am right, it should print [2,3,4,5,6]. Why does it print only [2,3,4,5]?
>

What you are asking about is "slicing".  x[1:5] returns everything
between index 1 through, but NOT including, index 5.  See

https://docs.python.org/3/tutorial/introduction.html#strings

which will give examples using strings.  A bit later the tutorial
addresses slicing in a list context.

BTW, the Python Tutorial is well worth reading in full!


-- 
boB


From __peter__ at web.de  Sat Dec 17 15:00:05 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 17 Dec 2016 21:00:05 +0100
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <o345g2$78b$1@blaine.gmane.org>

John wrote:

> Hi,
> 
>    I am new to Python, and I believe it's an easy question. I know R and
>    Matlab.
> 
> ************
>>>> x=[1,2,3,4,5,6,7]
>>>> x[0]
> 1
>>>> x[1:5]
> [2, 3, 4, 5]
> *************
> 
>     My question is: what does x[1:5] mean? By Python's convention, the
>     first element of a list is indexed as "0". Doesn't x[1:5] mean a
>     sub-list of x, indexed 1,2,3,4,5? If I am right, it should print
>     [2,3,4,5,6]. Why does it print only [2,3,4,5]?

Python uses half-open intervals, i. e. the first index is included, but the 
last index is not:

x[1:5] == [x[1], x[2], x[3], x[4]]

The advantage of this convention is that it allows easy splitting and length 
spefication.

>>> items
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]

To swap the head and tail:

>>> gap = 5
>>> items[gap:] + items[:gap]
[50, 60, 70, 80, 90, 0, 10, 20, 30, 40]

To extract a stride of a given length:

>>> start = 2
>>> length = 3
>>> items[start: start + length]
[20, 30, 40]

The disadvantage is that not everybody follows this convention...



From benjamin at python.org  Sat Dec 17 16:44:33 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 17 Dec 2016 13:44:33 -0800
Subject: [RELEASE] Python 2.7.13
Message-ID: <1482011073.1913762.822298361.75CAC690@webmail.messagingengine.com>

It is my pleasure to announce the release of Python 2.7.13, the latest
bugfix release of the venerable Python 2.7 series. This release
incorporates conservative bugfixes as well as improvements to keep
Python 2.7 running on modern systems.

The only change from the 2.7.13 release candidate 2 weeks ago is the
revert of a change that broke backwards compatibility with some rare C
extension patterns. See https://bugs.python.org/issue5322 for more
details.

Source archives and binaries are available at
    https://www.python.org/downloads/release/python-2713/

Please report bugs to our tracker:
    https://bugs.python.org/

2.7.14 will appear mid-2017.

All the best in the new year,
Benjamin Peterson
2.7 release manager


From tjreedy at udel.edu  Sat Dec 17 17:18:31 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 17 Dec 2016 17:18:31 -0500
Subject: python list index - an easy question
In-Reply-To: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <o34djk$77n$1@blaine.gmane.org>

On 12/17/2016 2:10 PM, John wrote:
> Hi,
>
>    I am new to Python, and I believe it's an easy question. I know R and Matlab.
>
> ************
>>>> x=[1,2,3,4,5,6,7]
>>>> x[0]
> 1
>>>> x[1:5]
> [2, 3, 4, 5]
> *************
>
>     My question is: what does x[1:5] mean?

The subsequence between slice positions 1 and 5, length 5-1=4.
Slice positions are before and after each item, not through them.
There are n+1 slice positions for n items: 0 before the first,
1 to n-1 between pairs, and n after the last.
Think of slice positions as tick marks on a line with the length 1
segment between as a cell holding a reference to one item.

  a b c d e
+-+-+-+-+-+
0 1 2 3 4 5

Slice 1:4 of length 3 is sequence with b, c, d.
Slice 3:3 of length 0 is an empty sequence.

> By Python's convention, the first element of a list is indexed as "0".

Think of 0 as .5 rounded down, or represent by the lower bound.
Other language round up to 1, or use the upper bound.

> Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5?

No.  It is items between 1:2, 2:3, 3:4, 4:5.


Terry Jan Reedy



From skybuck2000 at hotmail.com  Sat Dec 17 17:39:08 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Sat, 17 Dec 2016 14:39:08 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
Message-ID: <0f4d851b-4a2f-4628-bbb4-52bbcc999eed@googlegroups.com>

Unless you are capable of expressing problems in numerical terms you'll have very hard luck having it computed by a computer ! ;)

I did find some interesting docs about "decision trees".

Exhaustive search
Branch and Bound
Hill Climbing
Random/Best Effort Solutions

and so forth.

A computer programmer should be able to handle abstract descriptions as well like I gave in my initial posting.

What kind of "dressing" you want to give it is up to you, the dressing won't solve it though ! :)

Meanwhile I have also consider some kind of lookup table... or an intermediate table like you described...

Not yet sure if that will be of any help... (I am in doubt about the dynamic nature of the battles... perhaps an intermediate table would be wrong... or maybe it might be right, dynamic deaths vs staying alive inside intermediate table and such).

I also tried the static approach by multiplieing chances instead of subtracting the chances like damage done.

I also experimented with "Pascal/Delphi sets" in combination with arrays... this produced a very easy/simple permutation algorithm which consisted out of multiple for loops, which is kinda the obvious way of easily creating permutations... but it was still kinda interesting.

There is a problem with such loops though, also like the one you mention "for each" and "for in" and such... "under water" the compiler will loop the entire range of the type, and will use compare statements to see if it should enter the inner loop. So there is hidden overhead associated with it.

Thus such "generating" of permutations on the fly might have a hidden overhead cost associated with it, even if this was not the case, such for loop constructs will definetly have overheads in certain situations and this can quickly get out of hand and could even consume more processing time then the entire program.

I did also briefly considered this permutation loop, though it would require a lot of programming, 8x4=24 for loops, plus possible an additional attack loop.

You do have a little point that without a clear story it might be hard to understand the exact problem. I may consider describing the problem one more time, but it might still be with number or blue berries ! LOL :)


From bc at freeuk.com  Sat Dec 17 17:53:43 2016
From: bc at freeuk.com (BartC)
Date: Sat, 17 Dec 2016 22:53:43 +0000
Subject: python list index - an easy question
In-Reply-To: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <o34fj5$hhu$1@dont-email.me>

On 17/12/2016 19:10, John wrote:
> Hi,
>
>    I am new to Python, and I believe it's an easy question. I know R and Matlab.
>
> ************
>>>> x=[1,2,3,4,5,6,7]
>>>> x[0]
> 1
>>>> x[1:5]
> [2, 3, 4, 5]
> *************
>
>     My question is: what does x[1:5] mean?

x[A:B] means the slice consisting of x[A], x[A+1],... x[B-1]. (Although 
slices can shorter including those with be 0 or 1 elements.)

> By Python's convention, the first element of a list is indexed as "0".

Or the slice from the (A+1)th element to the B'th element inclusive, if 
you are informally using ordinal indexing (first, second, third etc).

> Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5?

Sublists and slices, once extracted, are indexed from 0 too.

Play around with some test code, but avoid test data containing numbers 
that are not too different from possible indices as that will be confusing!

Strings might be better:

   x = "ABCDEFGHIJKLM"

   print (x[1:5])

displays: BCDE

   print (x[1:5][0:2])        # slice of a slice

displays: BC

-- 
Bartc


From skybuck2000 at hotmail.com  Sat Dec 17 17:55:03 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Sat, 17 Dec 2016 14:55:03 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
Message-ID: <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>

Specially for Dennis, a nice story:

There are four little animals. The first animal is a tiger, the second animal is a deer, the third animal a parrot, the fourth animal is a fish.

The animals are very angry at each other and want to EAT each other ! =D

However each animal has a certain specialization. For example, the tiger has very sharp teeth, but can't hold it's breath for long under water.

The parrot has the ability to drive others insane with his chilping. The fish can lure other animals into the water in hopes of drowning them. 

And the deer can lure them into the swamp or make them fatigued from chasing them.

Fortunately for the four animals they don't need to EAT each other because another group of animals have arrived which are exactly like them.

So the four animals have decided to gang UP like negros in the GETTO ! LOL.

It's now GANG vs GANG.

But the poor little animals have run into a problem ?! Which of the other four animals should they EAT first ?! Or should they even attack multiple at the same time ?!

Should the tiger fight the tiger ?

Should the fish fight the fish ?

Should the parrot fight the parrot ?

Should the deer fight the deer ?

Or perhaps ?

Should the tiger eat the fish first ?

Should the fish try to drown the tiger first ?

Should the tiger and the fish gang up on the tiger ? But what about the enemy fish ? What will it do ?

Should all four animals attack the enemy tiger ? or another animal ?

Also for the gang to achieve victory all four enemy animals must be EATEN !

Now these animals wonder to themselfes ?

How should we attack the enemy ? Who should we attack first ? 

Every individual attack has a chance of success as given by the victory table. (A survival table could be calculated as well. Which would be the inverse of this).

Many possibilities for attack exists ?!

Which one do you recommend for them ?! 

Keep in mind that each animal is free to decide it's own attack plan. They do not need to agree with you.

Imagine yourself to be one of the animals.

Which attack strategy for yourself would be best to use to maximize your GANG of winning, no matter what other attack plans the animals have.

An example of a possible attack:


Red Tiger has his own attack plan eat in this order: Blue Tiger, Blue Fish, Blue Deer, Blue Parrot
Red Deer has his own attack plan eat in this order: Blue Fish, Blue Deer, Blue Parrot, Blue Tiger
Red Fish has his own attack plan eat in this order: Blue Fish, Blue Parrot, Blue Deer, Blue Tiger
Red Parrot has his own attack plan eat in this order: Blue Parrot, Blue Fish, Blue Deer, Blue Tiger

and vice versa for blue team ! ;) :)



From python at lucidity.plus.com  Sat Dec 17 20:01:10 2016
From: python at lucidity.plus.com (Erik)
Date: Sun, 18 Dec 2016 01:01:10 +0000
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
 <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
Message-ID: <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>

On 16/12/16 01:17, Chris Angelico wrote:
> On Fri, Dec 16, 2016 at 11:36 AM, Erik <python at lucidity.plus.com> wrote:
>> On 12/12/16 23:23, Chris Angelico wrote:
>>>
>>> In JavaScript, it's normal to talk about "calling a function as a
>>> constructor". When you do, there is a 'this' object before you start.
>>
>>
>> No there isn't. There is an implicit binding of a variable called "this"
>> based on the syntactic sugar of whether you're calling a function as method
>> on an object or not.
>>
>> In "strict" mode, [blah, blah, blah]

> I'm talking about when you call a function as a constructor: "new
> Foo()". Doesn't that have a 'this' object before the function starts?

Yes, in that case there is (I didn't grok that you meant using 'new' by 
"calling a function as a constructor", but it's obvious now you spell it 
out).

I wish I could find the resource I originally learned this stuff from, 
because it's quite enlightening and I'd like to link to it here - if one 
understands how things work generally under the covers it all makes much 
more sense, but I guess that's also a bad advert for a language (and why 
a lot of people get confused at first, and why it's a bit of a mess ;)).

But yes you're correct, in the case of using "new Func()" then "Func" is 
called with an implicit binding of 'this' that is to a newly created object.

Regards,
E.




From rosuav at gmail.com  Sat Dec 17 20:21:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 18 Dec 2016 12:21:56 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
 <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
 <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>
Message-ID: <CAPTjJmpazWZcs21FQw-d3E7HaZQd9=XjPUJFBy6BnuojkkjUtA@mail.gmail.com>

On Sun, Dec 18, 2016 at 12:01 PM, Erik <python at lucidity.plus.com> wrote:
> Yes, in that case there is (I didn't grok that you meant using 'new' by
> "calling a function as a constructor", but it's obvious now you spell it
> out).

Yeah. I thought that I got that terminology from MDN, but I can't find
it now, so it must have been from elsewhere.

> I wish I could find the resource I originally learned this stuff from,
> because it's quite enlightening and I'd like to link to it here - if one
> understands how things work generally under the covers it all makes much
> more sense, but I guess that's also a bad advert for a language (and why a
> lot of people get confused at first, and why it's a bit of a mess ;)).

Sounds like how Michael Schwern introduces his "Git for Ages 4 and Up"
talk - git may have a pretty poor UI, but has such beautiful innards
that it's best to understand it in that way. And he agrees that this
isn't how you _normally_ want to do things. When I explain Python, I
don't start by explaining the byte code interpreter or the operand
stack :)

ChrisA


From python at mrabarnett.plus.com  Sat Dec 17 21:34:49 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 18 Dec 2016 02:34:49 +0000
Subject: Unicode script
In-Reply-To: <c321b2ad-0615-212c-70d5-30f039d8765b@mrabarnett.plus.com>
References: <5852caa8$0$1605$c3e8da3$5496439d@news.astraweb.com>
 <392bb301-0e54-3607-7b7f-3c13e7bb6b9b@mrabarnett.plus.com>
 <o2v3k1$tqs$1@blaine.gmane.org>
 <c321b2ad-0615-212c-70d5-30f039d8765b@mrabarnett.plus.com>
Message-ID: <92769a61-c054-6c07-00f5-f05270b4b11d@mrabarnett.plus.com>

On 2016-12-16 02:44, MRAB wrote:
> On 2016-12-15 21:57, Terry Reedy wrote:
>> On 12/15/2016 1:06 PM, MRAB wrote:
>>> On 2016-12-15 16:53, Steve D'Aprano wrote:
>>>> Suppose I have a Unicode character, and I want to determine the script or
>>>> scripts it belongs to.
>>>>
>>>> For example:
>>>>
>>>> U+0033 DIGIT THREE "3" belongs to the script "COMMON";
>>>> U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN";
>>>> U+03BE GREEK SMALL LETTER XI "?" belongs to the script "GREEK".
>>>>
>>>>
>>>> Is this information available from Python?
>>>>
>>>>
>>>> More about Unicode scripts:
>>>>
>>>> http://www.unicode.org/reports/tr24/
>>>> http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt
>>>> http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt
>>>>
>>>>
>>> Interestingly, there's issue 6331 "Add unicode script info to the
>>> unicode database". Looks like it didn't make it into Python 3.6.
>>
>> https://bugs.python.org/issue6331
>> Opened in 2009 with patch and 2 revisions for 2.x.  At least the Python
>> code needs to be updated.
>>
>> Approved in principle by Martin, then unicodedata curator, but no longer
>> active.  Neither, very much, are the other 2 listed in the Expert's index.
>>
>>  From what I could see, both the Python API (there is no doc patch yet)
>> and internal implementation need more work.  If I were to get involved,
>> I would look at the APIs of PyICU (see Eryk Sun's post) and the
>> unicodescript module on PyPI (mention by Pander Musubi, on the issue).
>>
> For what it's worth, the post has prompted me to get back to a module I
> started which will report such Unicode properties, essentially the ones
> that the regex module supports. It just needs a few more tweaks and
> packaging up...
>
Finally completed and uploaded!

It's called 'uniprop' and it's at:

https://pypi.python.org/pypi/uniprop/1.0

For Python 3.4-3.6.



From alister.ware at ntlworld.com  Sun Dec 18 03:12:37 2016
From: alister.ware at ntlworld.com (alister)
Date: Sun, 18 Dec 2016 08:12:37 GMT
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <Vxr5A.331225$5e4.77889@fx27.am4>

On Sat, 17 Dec 2016 11:10:22 -0800, John wrote:

> Hi,
> 
>    I am new to Python, and I believe it's an easy question. I know R and
>    Matlab.
> 
> ************
>>>> x=[1,2,3,4,5,6,7]
>>>> x[0]
> 1
>>>> x[1:5]
> [2, 3, 4, 5] *************
> 
>     My question is: what does x[1:5] mean? By Python's convention, the
>     first element of a list is indexed as "0". Doesn't x[1:5] mean a
>     sub-list of x, indexed 1,2,3,4,5? If I am right, it should print
>     [2,3,4,5,6]. Why does it print only [2,3,4,5]?
> 
>    Thanks!!
> 
> John

as well as al the other excellent & detailed explanations
think of the slice working on the gaps between the elements (the ',') & 
not the element itself




-- 
In the long run we are all dead.
		-- John Maynard Keynes


From paul.christoph.goetze at gmail.com  Sun Dec 18 05:59:54 2016
From: paul.christoph.goetze at gmail.com (=?UTF-8?Q?Paul_G=c3=b6tze?=)
Date: Sun, 18 Dec 2016 11:59:54 +0100
Subject: python list index - an easy question
In-Reply-To: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
Message-ID: <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>

Hi John,

there is a nice short article by E. W. Dijkstra about why it makes sense
to start numbering at zero (and exclude the upper given bound) while
slicing a list. Might give a bit of additional understanding.

http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF

- paul


http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF


Am 17.12.2016 um 20:10 schrieb John:
> Hi, 
>
>    I am new to Python, and I believe it's an easy question. I know R and Matlab.
>
> ************
>>>> x=[1,2,3,4,5,6,7]
>>>> x[0]
> 1
>>>> x[1:5]
> [2, 3, 4, 5]
> *************
>
>     My question is: what does x[1:5] mean? By Python's convention, the first element of a list is indexed as "0". Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? If I am right, it should print [2,3,4,5,6]. Why does it print only [2,3,4,5]?
>
>    Thanks!!
>
> John




From elchino at cnn.cn  Sun Dec 18 07:25:21 2016
From: elchino at cnn.cn (ElChino)
Date: Sun, 18 Dec 2016 13:25:21 +0100
Subject: print() with no newline
Message-ID: <o35v4q$9o7$2@dont-email.me>

In this snippet:
  import sys
  PY3 = (sys.version_info[0] >= 3)

  def print_no_nl (s):
    if PY3:
      print (s, end="")
    else:
      print (s),

I'm getting a syntax error in Python2. Python3 is fine.
How can I make this Py2+3 compatible?


From kwpolska at gmail.com  Sun Dec 18 07:38:47 2016
From: kwpolska at gmail.com (Chris Warrick)
Date: Sun, 18 Dec 2016 13:38:47 +0100
Subject: print() with no newline
In-Reply-To: <o35v4q$9o7$2@dont-email.me>
References: <o35v4q$9o7$2@dont-email.me>
Message-ID: <CAMw+j7+ZYVu60=AGcmSaLczFtn6iVgC7qx49u=hY-4dZ+P=w3A@mail.gmail.com>

On 18 December 2016 at 13:25, ElChino <elchino at cnn.cn> wrote:
> In this snippet:
>   import sys
>   PY3 = (sys.version_info[0] >= 3)
>
>   def print_no_nl (s):
>     if PY3:
>       print (s, end="")
>     else:
>       print (s),
>
> I'm getting a syntax error in Python2. Python3 is fine.
> How can I make this Py2+3 compatible?

With a __future__ import, the Python 3 syntax will work with both Pythons:

from __future__ import print_function
print(s, end="")

-- 
Chris Warrick <https://chriswarrick.com/>
PGP: 5EAAEA16


From elchino at cnn.cn  Sun Dec 18 08:11:17 2016
From: elchino at cnn.cn (ElChino)
Date: Sun, 18 Dec 2016 14:11:17 +0100
Subject: print() with no newline
In-Reply-To: <mailman.155.1482064730.2322.python-list@python.org>
References: <o35v4q$9o7$2@dont-email.me>
 <CAMw+j7+ZYVu60=AGcmSaLczFtn6iVgC7qx49u=hY-4dZ+P=w3A@mail.gmail.com>
 <mailman.155.1482064730.2322.python-list@python.org>
Message-ID: <o361qu$i6h$1@dont-email.me>

Chris Warrick wrote:

>> I'm getting a syntax error in Python2. Python3 is fine.
>> How can I make this Py2+3 compatible?
> 
> With a __future__ import, the Python 3 syntax will work with both Pythons:
> 
> from __future__ import print_function
> print(s, end="")

Thanks. Lovely.




From bc at freeuk.com  Sun Dec 18 11:21:20 2016
From: bc at freeuk.com (BartC)
Date: Sun, 18 Dec 2016 16:21:20 +0000
Subject: python list index - an easy question
In-Reply-To: <mailman.156.1482070912.2322.python-list@python.org>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
Message-ID: <o36cvf$duq$1@dont-email.me>

On 18/12/2016 10:59, Paul G?tze wrote:
> Hi John,
>
> there is a nice short article by E. W. Dijkstra about why it makes sense
> to start numbering at zero (and exclude the upper given bound) while
> slicing a list. Might give a bit of additional understanding.
>
> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF

(This from somebody who apparently can't use a typewriter?!)

I don't know if the arguments there are that convincing. Both lower 
bounds of 0 and 1 are useful; some languages will use 0, some 1, and 
some can have any lower bound.

But a strong argument for using 1 is that in real life things are 
usually counted from 1 (and measured from 0).

So if you wanted a simple list giving the titles of the chapters in a 
book or on a DVD, on the colour of the front doors for each house in a 
street, usually you wouldn't be able to use element 0.

As for slice notation, I tend to informally use (not for any particulr 
language) A..B for an inclusive range, and A:N for a range of length N 
starting from A.

In Python you can also have a third operand for a range, A:B:C, which 
can mean that B is not necessarily one past the last in the range, and 
that the A <= i < B condition in that paper is no longer quite true.

In fact, A:B:-1 corresponds to A >= i > B, which I think is the same as 
condition (b) in the paper (but backwards), rather (a) which is favoured.

Another little anomaly in Python is that when negative indices are used, 
it suddenly switches to 1-based indexing! Or least, when -index is 
considered:

   x = [-4,-3,-2,-1]

   print x[-1]       # -1  Notice the correspondence here...
   print x[-2]       # -2

   x = [1, 2, 3, 4]

   print x[1]        # 2   ...and the lack of it here
   print x[2]        # 3


-- 
Bartc



From alister.ware at ntlworld.com  Sun Dec 18 15:44:36 2016
From: alister.ware at ntlworld.com (alister)
Date: Sun, 18 Dec 2016 20:44:36 GMT
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
Message-ID: <UyC5A.407302$yH3.354840@fx40.am4>

On Sun, 18 Dec 2016 16:21:20 +0000, BartC wrote:

> On 18/12/2016 10:59, Paul G?tze wrote:
>> Hi John,
>>
>> there is a nice short article by E. W. Dijkstra about why it makes
>> sense to start numbering at zero (and exclude the upper given bound)
>> while slicing a list. Might give a bit of additional understanding.
>>
>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
> 
> (This from somebody who apparently can't use a typewriter?!)
> 
> I don't know if the arguments there are that convincing. Both lower
> bounds of 0 and 1 are useful; some languages will use 0, some 1, and
> some can have any lower bound.
> 
> But a strong argument for using 1 is that in real life things are
> usually counted from 1 (and measured from 0).
> 
> So if you wanted a simple list giving the titles of the chapters in a
> book or on a DVD, on the colour of the front doors for each house in a
> street, usually you wouldn't be able to use element 0.
> 
> As for slice notation, I tend to informally use (not for any particulr
> language) A..B for an inclusive range, and A:N for a range of length N
> starting from A.
> 
> In Python you can also have a third operand for a range, A:B:C, which
> can mean that B is not necessarily one past the last in the range, and
> that the A <= i < B condition in that paper is no longer quite true.
> 
> In fact, A:B:-1 corresponds to A >= i > B, which I think is the same as
> condition (b) in the paper (but backwards), rather (a) which is
> favoured.
> 
> Another little anomaly in Python is that when negative indices are used,
> it suddenly switches to 1-based indexing! Or least, when -index is
> considered:
> 
>    x = [-4,-3,-2,-1]
> 
>    print x[-1]       # -1  Notice the correspondence here...
>    print x[-2]       # -2
> 
>    x = [1, 2, 3, 4]
> 
>    print x[1]        # 2   ...and the lack of it here print x[2]       
>    # 3


as I said earlier
take the indicates as being the spaces between the elements & it makes 
much more sense


-- 
falsie salesman, n:
	Fuller bust man.


From torriem at gmail.com  Sun Dec 18 16:04:56 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sun, 18 Dec 2016 14:04:56 -0700
Subject: python list index - an easy question
In-Reply-To: <o36cvf$duq$1@dont-email.me>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
Message-ID: <4f99bfd5-6391-a0b4-bf23-c84622bf1358@gmail.com>

On 12/18/2016 09:21 AM, BartC wrote:
> On 18/12/2016 10:59, Paul G?tze wrote:
>> Hi John,
>>
>> there is a nice short article by E. W. Dijkstra about why it makes sense
>> to start numbering at zero (and exclude the upper given bound) while
>> slicing a list. Might give a bit of additional understanding.
>>
>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
> 
> (This from somebody who apparently can't use a typewriter?!)
> 
> I don't know if the arguments there are that convincing. Both lower 
> bounds of 0 and 1 are useful; some languages will use 0, some 1, and 
> some can have any lower bound.
> 
> But a strong argument for using 1 is that in real life things are 
> usually counted from 1 (and measured from 0).
> 
> So if you wanted a simple list giving the titles of the chapters in a 
> book or on a DVD, on the colour of the front doors for each house in a 
> street, usually you wouldn't be able to use element 0.

It also depends on whether you want to number the spaces between the
objects or the objects themselves. To use your DVD example, the first
chapter will probably be starting at time zero, not time one.

In another example, babies start out at "zero" years old not "one."  But
at the same time we refer the first year of life.  Maybe it's not a
phrase much used these days but it used to be common to say something
like "in my 15th year," meaning when I was 14.  Maybe a more common use
would be "the first year of my employment at this company."

I'm not sure it makes sense to having slicing be zero-based but indexing
itself be 1-based, but I think a case could have been made (though I'm
glad it was not).


From none at invalid.com  Sun Dec 18 16:37:38 2016
From: none at invalid.com (mm0fmf)
Date: Sun, 18 Dec 2016 21:37:38 +0000
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <ebg4apF76f9U1@mid.individual.net>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
 <ebg4apF76f9U1@mid.individual.net>
Message-ID: <o36vge$kr1$2@dont-email.me>

On 15/12/2016 18:05, Peter Pearson wrote:
> On Wed, 14 Dec 2016 11:50:30 -0600, Skip Montanaro wrote:
>> On Wed, Dec 14, 2016 at 11:40 AM, Peter Pearson
>> <pkpearson at nowhere.invalid> wrote:
>>> Train your fingers to use C-[.
>>
>> As I recall, the location of the Ctrl key was one of the differences
>> between Sun and PC101 keyboards. Doesn't matter so much now, as Sun
>> has gone the way of the dodo, but it moved around more for me than ESC
>> over the years.
>
> Absolutely right.  Random migrations of the Ctrl key annoyed so many
> of us set-in-our-ways geezers that Linux distributions always seem to
> come with an easily activated option to put the Ctrl key where it
> belongs, namely to the left of the A, right where God put it on Adam's
> ASR 33.
>
+1 for knowing where CTRL should be.
Bonus +1 for having used an ASR33.

;-)


From cs at zip.com.au  Sun Dec 18 16:56:16 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 19 Dec 2016 08:56:16 +1100
Subject: python list index - an easy question
In-Reply-To: <o36cvf$duq$1@dont-email.me>
References: <o36cvf$duq$1@dont-email.me>
Message-ID: <20161218215616.GA82242@cskk.homeip.net>

On 18Dec2016 16:21, BartC <bc at freeuk.com> wrote:
>On 18/12/2016 10:59, Paul G?tze wrote:
>>there is a nice short article by E. W. Dijkstra about why it makes sense
>>to start numbering at zero (and exclude the upper given bound) while
>>slicing a list. Might give a bit of additional understanding.
>>
>>http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
>
>(This from somebody who apparently can't use a typewriter?!)
>
>I don't know if the arguments there are that convincing. Both lower 
>bounds of 0 and 1 are useful; some languages will use 0, some 1, and 
>some can have any lower bound.

0 makes a lot of arithmetic simpler if you think of the index as the offset 
from the start of the array/list.

>But a strong argument for using 1 is that in real life things are 
>usually counted from 1 (and measured from 0).

Shrug. Yep. But again, if you visualise the index as an offset (== "measure") 
it is a natural fit.

>Another little anomaly in Python is that when negative indices are used, it 
>suddenly switches to 1-based indexing! Or least, when -index is considered:

Not if you consider it to count from the range end. So range 0:5 (which in 
Python includes indices 0,1,2,3,4); index -1 places you at 5-1 ==> 4, which is 
consistent. Again, this makes a lot of the arithmetic simpler.

See sig quote for another example of a python style range: birth to death.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

There's no need to worry about death, it will not happen in your lifetime.
        - Raymond Smullyan


From skip.montanaro at gmail.com  Sun Dec 18 16:59:42 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sun, 18 Dec 2016 15:59:42 -0600
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <o36vge$kr1$2@dont-email.me>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
 <ebg4apF76f9U1@mid.individual.net> <o36vge$kr1$2@dont-email.me>
Message-ID: <CANc-5UwpyhQtH+csSzaaa7p_hd8Ozs6YZA42+fvctwit_cwQSQ@mail.gmail.com>

> +1 for knowing where CTRL should be.
Bonus +1 for having used an ASR33.

;-)


I'm sure I must have used an ASR33, but can't recall what it might have
been connected to. I do remember using card punch machines for IBM 360
input in 1972 at USC, and toggling front panel switches for binary input on
a PDP-11 in one engineering class in grad school at Iowa. Hand assembling
your program isn't terrific fun.

With this polar vortex going on, I'm not sure when I'll have the cojones to
brave the weather for an otherwise unnecessary trip to the Apple Store.

Skip


From greg.ewing at canterbury.ac.nz  Sun Dec 18 17:03:50 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 19 Dec 2016 11:03:50 +1300
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <o36vge$kr1$2@dont-email.me>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
 <ebg4apF76f9U1@mid.individual.net> <o36vge$kr1$2@dont-email.me>
Message-ID: <ebofe8F7uovU1@mid.individual.net>

mm0fmf wrote:
> +1 for knowing where CTRL should be.
> Bonus +1 for having used an ASR33.

And it's quite remarkable that the designers of the ASR33
knew exactly where it would need to be for Emacs users
years later! I think Richard Stallman must have a time
machine as well.

-- 
Greg


From skybuck2000 at hotmail.com  Sun Dec 18 17:06:00 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Sun, 18 Dec 2016 14:06:00 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
 <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
Message-ID: <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>

Dennis wrote:

<snip>

"
	Instead you /now/ have ONE set of R marching down FOUR sets of B

	RT		RD		RF		RP	<-		attackers
	BT		BF		BF		BP			round 1
	BF		BD		BP		BF			round 2
	BD		BP		BD		BD			round 3
	BP		BT		BT		BT			round 4		
"

Yes this is how the problem works.

Also keep in mind that an attack is only valid if the target is still alive, otherwise the attacker would move to the next one.

So pre-computing an attack plan/outcome or storing it might not be so usefull for on color, since the other color might already be dead and thus attack plan was calculated incorrectly potentially.

So it's quite a difficult/complex/dynamic problem !


From bc at freeuk.com  Sun Dec 18 17:21:01 2016
From: bc at freeuk.com (BartC)
Date: Sun, 18 Dec 2016 22:21:01 +0000
Subject: python list index - an easy question
In-Reply-To: <mailman.159.1482095110.2322.python-list@python.org>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <4f99bfd5-6391-a0b4-bf23-c84622bf1358@gmail.com>
 <mailman.159.1482095110.2322.python-list@python.org>
Message-ID: <o3721s$upl$1@dont-email.me>

On 18/12/2016 21:04, Michael Torrie wrote:
> On 12/18/2016 09:21 AM, BartC wrote:

>> So if you wanted a simple list giving the titles of the chapters in a
>> book or on a DVD, on the colour of the front doors for each house in a
>> street, usually you wouldn't be able to use element 0.
>
> It also depends on whether you want to number the spaces between the
> objects or the objects themselves. To use your DVD example, the first
> chapter will probably be starting at time zero, not time one.
>
> In another example, babies start out at "zero" years old not "one."  But
> at the same time we refer the first year of life.  Maybe it's not a
> phrase much used these days but it used to be common to say something
> like "in my 15th year," meaning when I was 14.  Maybe a more common use
> would be "the first year of my employment at this company."

There's the fence analogy (perhaps similar to what alister said):

You have a fence made up of one-metre-wide panels that fit between two 
posts.

For a 10-metre fence, you need 11 posts, and 10 panels.

The posts can conveniently be numbered from 0 to 11, as that also gives 
you the distance of each one from the start of the fence.

But posts are thin. Panels are wide, and they might as well be 
conventionally numbered from 1, as you can't use the panel number to 
tell you how far it is from the start (is it the left bit of the panel 
or the right bit?).

Panels obviously correspond to the data in each list element; posts are 
harder to place, except perhaps as alister's commas (but then there have 
to be extra invisible commas at each end).

(The fence needs to divide an open area not surround an enclosed space, 
as otherwise the analogy breaks down; you will have N panels and N posts!)

> I'm not sure it makes sense to having slicing be zero-based but indexing
> itself be 1-based, but I think a case could have been made (though I'm
> glad it was not).

They need to be the same.

(Zero-based necessarily has to be used with offsets from pointers from 
example. In C, array indexing is inextricably tied up with 
pointer/offset arithmetic, so indexing /has/ to be zero-based.

But that doesn't apply in other languages where the choice could have 
been different.)

-- 
Bartc


From bc at freeuk.com  Sun Dec 18 17:36:00 2016
From: bc at freeuk.com (BartC)
Date: Sun, 18 Dec 2016 22:36:00 +0000
Subject: python list index - an easy question
In-Reply-To: <o3721s$upl$1@dont-email.me>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <4f99bfd5-6391-a0b4-bf23-c84622bf1358@gmail.com>
 <mailman.159.1482095110.2322.python-list@python.org>
 <o3721s$upl$1@dont-email.me>
Message-ID: <o372tu$23a$1@dont-email.me>

On 18/12/2016 22:21, BartC wrote:
> On 18/12/2016 21:04, Michael Torrie wrote:
>> On 12/18/2016 09:21 AM, BartC wrote:
>
>>> So if you wanted a simple list giving the titles of the chapters in a
>>> book or on a DVD, on the colour of the front doors for each house in a
>>> street, usually you wouldn't be able to use element 0.
>>
>> It also depends on whether you want to number the spaces between the
>> objects or the objects themselves. To use your DVD example, the first
>> chapter will probably be starting at time zero, not time one.
>>
>> In another example, babies start out at "zero" years old not "one."  But
>> at the same time we refer the first year of life.  Maybe it's not a
>> phrase much used these days but it used to be common to say something
>> like "in my 15th year," meaning when I was 14.  Maybe a more common use
>> would be "the first year of my employment at this company."
>
> There's the fence analogy (perhaps similar to what alister said):
>
> You have a fence made up of one-metre-wide panels that fit between two
> posts.
>
> For a 10-metre fence, you need 11 posts, and 10 panels.
>
> The posts can conveniently be numbered from 0 to 11,

... 0 to 10.

That's the thing with zero-based; it might reduce some off-by-one errors 
but could introduce others.

With the panels you have 10 panels numbered 1 to 10; what could be 
simpler or more intuitive?

-- 
Bartc


From python at lucidity.plus.com  Sun Dec 18 17:52:09 2016
From: python at lucidity.plus.com (Erik)
Date: Sun, 18 Dec 2016 22:52:09 +0000
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <CAPTjJmpazWZcs21FQw-d3E7HaZQd9=XjPUJFBy6BnuojkkjUtA@mail.gmail.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
 <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
 <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>
 <CAPTjJmpazWZcs21FQw-d3E7HaZQd9=XjPUJFBy6BnuojkkjUtA@mail.gmail.com>
Message-ID: <8c339250-848e-5155-fa5d-c826d81b9da1@lucidity.plus.com>


NOTE: If you found this message by searching for help on how Python 
works, be aware that it's discussing how JavaScript works, not Python! 
Look elsewhere :)

Chris, this isn't directed at you (I think you get it) - just following 
up with some detail for anyone who might discover this sub-thread when 
searching in future.

On 18/12/16 01:21, Chris Angelico wrote:
> On Sun, Dec 18, 2016 at 12:01 PM, Erik <python at lucidity.plus.com> wrote:
>> I wish I could find the resource I originally learned this stuff from,
>> because it's quite enlightening and I'd like to link to it here

> Sounds like how Michael Schwern introduces his "Git for Ages 4 and Up"
> talk

Just in case anyone is remotely interested, I've found a reasonable link 
(but it's not the one I was looking for).

I realise this is off-topic for a Python list, but it has come up 
because of a Python question so I thought why not ... (I wish someone 
had explained this to me in roughly-Python terms when I first had to 
deal with JS ;)).

In short, AIUI, all JS functions have an implicit initial parameter 
which is named 'this'. What that parameter is bound to depends on the 
context of the call to that function object. In Python the callable is 
of a particular type (method, function, generator) but on JS (for the 
purposes of this discussion (pun intended ;)) they are all equal and 
it's the calling context that determines what happens.

A JS function, whether declared with "function Foo(a, b, c) {}" or as an 
expression "function (a, b, c) {}" has an implicit initial parameter 
which is bound to the variable 'this' (so they are really "function 
Foo(this, a, b, c) {}", "function (this, a, b, c) {}").

The value of that initial first parameter is determined by how they are 
called. There are three main ways:

1) Method call:
    "obj.foo(1, 2, 3)" is syntactic sugar for "obj.foo(obj, 1, 2, 3)".

2) Function call:
    "foo(1, 2, 3)" is syntactic sugar for "foo(GLOBAL_OBJECT, 1, 2, 3)" 
where "GLOBAL_OBJECT" is whatever the execution environment defines as 
being the global object. In a browser, I think it's "window", in the JS 
strict mode, I think it's "undefined" (i.e., no object), in Node.js I 
think it might be something else entirely.

3) Constructor call:
    "new foo(1, 2, 3)" is syntactic sugar for "foo(NEWOBJ, 1, 2, 3)" 
where NEWOBJ is a newly allocated, empty object.

The other way of invoking a function object is via its "call" method 
where the initial 'this' parameter can be explicitly given - 
"foo.call(SPAM, 1, 2, 3)" - which is how to implement bound methods and 
that sort of thing.


The main area of confusion seems to be when passing function expressions 
(closures) as a callback to another function:

function Foo() {
   ham(this.a, this.b, function (x) { this.frobnicate(x); });
}

Let's assume this is called with "obj.Foo();". In the function itself, 
'this', is bound to 'obj'. However, in the body of the (anonymous) 
function expression/closure (which sort of looks to be at the same scope 
as everything else) 'this' is bound according to the rules above being 
applied to at the point it is called - the object it is bound to almost 
certainly doesn't have the "frobnicate" method expected.

This is why the following works:

function Foo() {
   _this = this;
   ham(this.a, this.b, function (x) { _this.frobnicate(x); });
}

The closure now knows how to address the 'this' object for the original 
method even though it is called later by whatever 'ham()' does. Using a 
bound function as the callback works in the same way.



Anyway, off-topic as I said, but if it helps other Pythoneers get to 
grips with some of the weird JS semantics, it's all good :)



The link I *did* find (which probably has a bit more depth) is:

https://rainsoft.io/gentle-explanation-of-this-in-javascript/


Cheers, E.


From rosuav at gmail.com  Sun Dec 18 18:04:44 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 19 Dec 2016 10:04:44 +1100
Subject: The right way to 'call' a class attribute inside the same class
In-Reply-To: <8c339250-848e-5155-fa5d-c826d81b9da1@lucidity.plus.com>
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
 <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
 <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>
 <CAPTjJmpazWZcs21FQw-d3E7HaZQd9=XjPUJFBy6BnuojkkjUtA@mail.gmail.com>
 <8c339250-848e-5155-fa5d-c826d81b9da1@lucidity.plus.com>
Message-ID: <CAPTjJmogyFm0iNegO1xkJdKC+nRTHXyrpZxtUm-2xbM4dTnnMg@mail.gmail.com>

On Mon, Dec 19, 2016 at 9:52 AM, Erik <python at lucidity.plus.com> wrote:
>
> 1) Method call:
>    "obj.foo(1, 2, 3)" is syntactic sugar for "obj.foo(obj, 1, 2, 3)".

And the bit you have to be REALLY careful of when working with both
Python and JS is that you have to have "obj.foo(...)" as a single
expression. Consider:

# Python
lst = []
func = lst.append
func(1); func(2); func(3)
# lst now holds [1,2,3]

// JavaScript
var arr = []
var func = arr.push.bind(arr) // IMPORTANT
func(1); func(2); func(3)
// arr now contains [1,2,3]

If you just take "arr.push", you don't get a bound method. You get an
unbound method. You have to explicitly bind that back to the array
object, or its "this" pointer won't be set.

This is so important (eg with callback functions) that JS introduced a
new type of function to help deal with the problem...

ChrisA


From ben.usenet at bsb.me.uk  Sun Dec 18 20:10:55 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Mon, 19 Dec 2016 01:10:55 +0000
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
Message-ID: <87zijs4tb4.fsf@bsb.me.uk>

BartC <bc at freeuk.com> writes:

> On 18/12/2016 10:59, Paul G?tze wrote:
>> there is a nice short article by E. W. Dijkstra about why it makes sense
>> to start numbering at zero (and exclude the upper given bound) while
>> slicing a list. Might give a bit of additional understanding.
>>
>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
>
> (This from somebody who apparently can't use a typewriter?!)
>
> I don't know if the arguments there are that convincing. Both lower
> bounds of 0 and 1 are useful; some languages will use 0, some 1, and
> some can have any lower bound.
>
> But a strong argument for using 1 is that in real life things are
> usually counted from 1 (and measured from 0).

The index of an element is a measure, not a count.

<snip>
-- 
Ben.


From metal.suomi at gmail.com  Mon Dec 19 01:21:54 2016
From: metal.suomi at gmail.com (metal.suomi at gmail.com)
Date: Sun, 18 Dec 2016 22:21:54 -0800 (PST)
Subject: read a table and make a basic plot
Message-ID: <2427726a-b37c-409c-a850-56fdf75da036@googlegroups.com>

Hi, I'm learning python and full of extensive tutorials around. Getting a bit lost and overflowed in my head with tuples, dictionaries, lists, etc ... etc... Everything great, but I'd like to perform some basic task while learning the rest. For example, I'm having a hard time to find some practical info to get me up to speed. E.g. doing the following list of tasks:

- read a file named 'data.dat' (let's suppose it has 13 columns and n lines) and read column 3 and 4

-plot column 3 vs column 4

-write a file 'outcome.txt' which has column 3 and 4

How do I do read and write of files? Is there some extra library I need for doing plots? I have read about  matplotlib.pyplot. How can I do the plot above using this library? (running python3.5 in OSX).

Thanks!
J.


From __peter__ at web.de  Mon Dec 19 04:01:46 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 19 Dec 2016 10:01:46 +0100
Subject: read a table and make a basic plot
References: <2427726a-b37c-409c-a850-56fdf75da036@googlegroups.com>
Message-ID: <o387lm$gg0$1@blaine.gmane.org>

metal.suomi at gmail.com wrote:

> Hi, I'm learning python and full of extensive tutorials around. Getting a
> bit lost and overflowed in my head with tuples, dictionaries, lists, etc
> ... etc... Everything great, but I'd like to perform some basic task while
> learning the rest. For example, I'm having a hard time to find some
> practical info to get me up to speed. E.g. doing the following list of
> tasks:
> 
> - read a file named 'data.dat' (let's suppose it has 13 columns and n
> lines) and read column 3 and 4
> 
> -plot column 3 vs column 4
> 
> -write a file 'outcome.txt' which has column 3 and 4
> 
> How do I do read and write of files? Is there some extra library I need
> for doing plots? I have read about  matplotlib.pyplot. How can I do the
> plot above using this library? (running python3.5 in OSX).
> 
> Thanks!
> J.

Here's a simple implementation that assumes both input and output file

$ cat plot_data.py
import csv
import sys

from matplotlib import pyplot

# get file names and column indices from commandline arguments
infile = sys.argv[1]
outfile = sys.argv[2]
xindex = int(sys.argv[3]) - 1
yindex = int(sys.argv[4]) - 1

# read the specified input file columns into separate lists
x = []
y = []
with open(infile) as instream:
    for row in csv.reader(instream, delimiter="\t"):
        x.append(float(row[xindex]))
        y.append(float(row[yindex]))

# write specified columns to output file
with open(outfile, "w") as outstream:
    csv.writer(outstream, delimiter="\t").writerows(zip(x, y))

# draw and show plot
pyplot.plot(x, y)
pyplot.show()

Use it like this:

$ python3 plot_data.py data.dat outfile.txt 3 4


PS: I used the following script to create some sample data:

import math
import random

ROWS = 100

def make_func(i):
    if i == 2:
        return lambda i: i
    else:
        N = random.randrange(1, 30)
        def f(i):
            return math.sin(i * N / ROWS)
        return f


column_funcs = [make_func(i) for i in range(13)]

with open("data.dat", "w") as outstream:
    for i in range(ROWS):
        print(
            "\t".join(map(str, [g(i) for g in column_funcs])),
            file=outstream
        )




From __peter__ at web.de  Mon Dec 19 04:34:11 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 19 Dec 2016 10:34:11 +0100
Subject: read a table and make a basic plot
References: <2427726a-b37c-409c-a850-56fdf75da036@googlegroups.com>
 <o387lm$gg0$1@blaine.gmane.org>
Message-ID: <o389ie$lsr$1@blaine.gmane.org>

Peter Otten wrote:

> Here's a simple implementation that assumes both input and output file

... are in TAB-delimited text format.




From skybuck2000 at hotmail.com  Mon Dec 19 04:34:42 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Mon, 19 Dec 2016 01:34:42 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
 <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
 <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>
Message-ID: <4c5af126-ab13-441d-bc60-212ad89c5d98@googlegroups.com>

Skybuck wrote:

"
Also keep in mind that an attack is only valid if the target is still alive, otherwise the attacker would move to the next one.
 
So pre-computing an attack plan/outcome or storing it might not be so usefull for on color, since the other color might already be dead and thus attack plan was calculated incorrectly potentially.
"

^- Apperently this text/posting was essential/the key for Skybuck's Magical Brain to come up with a solution !

I love how Skybuck's Magical Brain comes up with solutions after some sleep ! =D

Just need to input the problem before sleeping, then in the morning ! TA-TA: Solution ! =D

I just woke up, thought a little bit and then came up with the following solution !

The problem with the lookup/intermediate table is ALIVE/DEATH of own object (also teammate objects), furthermore another problem is ALIVE/DEATH of enemy object(s).

So now the solution is obvious ! A special lookup table as follows:

(Furthermore I just had another idea to compute an enemy health reduction or perhaps enemy health outcome, both are possible ! ;) Yeah... enemy health calculation probably possible and accurate, since alive/death of enemy also known on input and thus result can be calculated/stored in table ! ;)

EnemyHealth (could also be EnemyHealthReduction if that were necessary for some reason ! ;) total damages would be summed, but probably not necessary to do it that way).


Enemy1Health,Enemy2Health,Enemy3Health,Enemy4Health =
Friendly1AliveStatus, Friendly2AliveStatus, Friendly3AliveStatus, Friendly4AliveStatus, Object1AttackPermutations, Object2AttackPermutations, Object3AttackPermutations,Object4AttackPermutations,Enemy1AliveStatus,Enemy2AliveStatus,Enemy3AliveStatus,Enemy4AliveStatus

^ So what I just tried to describe above is an 16 dimensional array as follows:

2x2x2x2x24x24x24x24x2x2x2x2

So computational complexity added by alive/death status is only x8 which is totally great... total complexity is therefore:

24x24x24x24x8

(For each attack the lookup table can be reused so that perfect too)

So the initial complexity of 24x24x24x24x24x24x24x24x4(?) has now been reduced too:

1x24x24x24x24x8 (computations)

It will probably still be necessary to "compute/lookup" all possibilities just to be sure, however doing an 16 way array lookup should be way faster than computing while loops and if statements and what not... though the table is quite large so some random access may occur.

16 random access is roughly 16x100 nanoseconds. Plus some 5 minute looping overhead or so so let's calculate rough indication of new processing time.

24x24x24x24x24x24x24x24x4(?) (look ups)


But first I try and calculate some kind of indicator of current per loop processing time.

24^8 = 110075314176 battles / 80.000 seconds = 1375941.4272 battles per second.

The 5 minute overhead is about 300 seconds so seems minimal not going to adjust for that for now.

Now let's compute new expected running time.

Perhaps crazyly inaccurate calculation since it does not account for data cache hits... but just worst case scenerio or something:

It will be at least 16x100 nanoseconds, then 8 branches or so to determine winner is about 8 x branch time which is roughly 15 cycle clocks of 2 gigahertz or so. Then also 8 data field fetches another 8x1 cycle, and perhaps 8 assignments or so to store healths so roughly 15+8+8 = 13 plus probably some while loop to repeat attacks until there is a winner though this was already accounted for in the branches above... so for now I'll stick with 31 cycles or so

So 31 cycles of 2.000.000.000 hz machine I would guess is roughly: 

0.0000000155 seconds, roughly 15.5 nanoseconds so worst case scenerio:

16x100 = 1600 + 15 = 1615.5 nanoseconds to compute/lookup a battle.

Number of battles: 110075314176 x 1615.5 = 177826670051328 nanoseconds

= 177826.670051328 seconds.

According to my initial calculations it will be far worse off lol !

However the 8x dimensions could be put close together so that at least those are cached, then perhaps a lookup will only take 10 nanoseconds or something or maybe even less.

So then it becomes 8x10 nanoseconds + 8*100 nanoseconds = 880 nanoseconds

Seems like roughly a reduction... then I am back to square one roughly running time of 90.000 seconds. Plus ofcourse some pre-compute overhead...

Though I am kinda bad at estimating running time like this... 

I am hopefully that the look ups will proceed much faster...

Sometimes of those 8x100 nanosecond seeks there will be some L1/L2 data cache hits hopefully so that should accelerate it somewhat.

What actually needs to be stored is enemy health... this can be done with one byte... 100 is near 128 bits... so at least 7 bits needed for 100, so only 1 bit could be saved so doesn't really seem worth it to save 4 bits in total ! ;)

The pre-compute time for the table lookups should be quite fast...

The main processing for generating the lookup is:

24^4 = 331776 x 8 = 2.654.208

That's nothing compared to the 110.000.000.000 that it used to be ! ;) :)

So I am very hopefull that I have found a nice data acceleration structure ! ;) :)

Having this conversation with you was usefull, thanks !

Bye,
  Skybuck =D


From bc at freeuk.com  Mon Dec 19 05:26:56 2016
From: bc at freeuk.com (BartC)
Date: Mon, 19 Dec 2016 10:26:56 +0000
Subject: python list index - an easy question
In-Reply-To: <87zijs4tb4.fsf@bsb.me.uk>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
Message-ID: <o38ciu$9t4$1@dont-email.me>

On 19/12/2016 01:10, Ben Bacarisse wrote:
> BartC <bc at freeuk.com> writes:
>
>> On 18/12/2016 10:59, Paul G?tze wrote:
>>> there is a nice short article by E. W. Dijkstra about why it makes sense
>>> to start numbering at zero (and exclude the upper given bound) while
>>> slicing a list. Might give a bit of additional understanding.
>>>
>>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
>>
>> (This from somebody who apparently can't use a typewriter?!)
>>
>> I don't know if the arguments there are that convincing. Both lower
>> bounds of 0 and 1 are useful; some languages will use 0, some 1, and
>> some can have any lower bound.
>>
>> But a strong argument for using 1 is that in real life things are
>> usually counted from 1 (and measured from 0).
>
> The index of an element is a measure, not a count.

You need to take your C hat off, I think.

-- 
Bartc


From ned at nedbatchelder.com  Mon Dec 19 07:32:35 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 19 Dec 2016 04:32:35 -0800 (PST)
Subject: python list index - an easy question
In-Reply-To: <o36cvf$duq$1@dont-email.me>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
Message-ID: <9ddec06f-3b92-4986-92b8-34eb6aa675e6@googlegroups.com>

On Sunday, December 18, 2016 at 11:21:38 AM UTC-5, BartC wrote:
> On 18/12/2016 10:59, Paul G?tze wrote:
> > Hi John,
> >
> > there is a nice short article by E. W. Dijkstra about why it makes sense
> > to start numbering at zero (and exclude the upper given bound) while
> > slicing a list. Might give a bit of additional understanding.
> >
> > http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
> 
> (This from somebody who apparently can't use a typewriter?!)

Don't judge a book by its rendering technology :)   Dijkstra was
"one of the most influential members of computing science's founding
generation": https://en.wikipedia.org/wiki/Edsger_W._Dijkstra

--Ned.


From random832 at fastmail.com  Mon Dec 19 07:57:33 2016
From: random832 at fastmail.com (Random832)
Date: Mon, 19 Dec 2016 07:57:33 -0500
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <ebofe8F7uovU1@mid.individual.net>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5Uwcn0A2P3ZEo4qw0E6S1kVgfn-VL+kiE-roae3uW4Gp6A@mail.gmail.com>
 <CANc-5UyJa657tKkPSmKkOzCP9RkpDChwanytHhX4XFYLLMWqmA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
 <ebg4apF76f9U1@mid.individual.net> <o36vge$kr1$2@dont-email.me>
 <ebofe8F7uovU1@mid.individual.net>
Message-ID: <1482152253.1697031.823531193.25AC26DA@webmail.messagingengine.com>

On Sun, Dec 18, 2016, at 17:03, Gregory Ewing wrote:
> mm0fmf wrote:
> > +1 for knowing where CTRL should be.
> > Bonus +1 for having used an ASR33.
> 
> And it's quite remarkable that the designers of the ASR33
> knew exactly where it would need to be for Emacs users
> years later! I think Richard Stallman must have a time
> machine as well.

Except for the fact that the actual keyboard that Emacs was originally
developed for [the Knight Keyboard, and the later Symbolics "Space
Cadet" Keyboards] had the control key more or less where it is on modern
PC keyboards [slightly further to the right, so easier to reach with the
thumb of the same-side hand], and the Meta key to its left. To the left
of A was "rub out".

I assume the original Emacs users used proper touch-typing style:
modifier keys with the opposite hand from the letter they were using it
with, rather than trying to contort the same-side hand into hitting both
keys at once. Meanwhile, users of Unix systems were stuck with terminals
like the ADM-3A and VT-100, which, inherited from the ASR-33 but one
must assume the real reason was to save money, only had one control key
and no true meta key (one escape key). I suspect there would have only
been one shift key if they could have got away with it.


From PointedEars at web.de  Mon Dec 19 08:16:44 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Mon, 19 Dec 2016 14:16:44 +0100
Subject: The right way to 'call' a class attribute inside the same class
References: <CAAp0bGsw+xKc30h1Ya4n3EAEgibvWgagDpyvbC5-jmNaBY86VA@mail.gmail.com>
 <mailman.16.1481487105.2322.python-list@python.org>
 <3924532.8F6SAcFxjW@PointedEars.de>
 <CAAp0bGvtxqt9c+TxQ9=wro1ZT0vBWpW59+6gTz-n8a-+=QyjpQ@mail.gmail.com>
 <mailman.27.1481565498.2322.python-list@python.org>
 <95604cee-a78d-4e09-9dcc-ed163ea67db0@googlegroups.com>
 <8560mo20wg.fsf@benfinney.id.au>
 <CAPTjJmp8yomjkbhjzKjHjAKOhDv5q2tdM4Z2A3qCzBJEFYhpLg@mail.gmail.com>
 <b351a378-426f-9e02-d701-3b2eae494fee@lucidity.plus.com>
 <CAPTjJmosK304Yg93-XYVt5aQFODYjPtsh0TyxbdFzvOhy+HZsg@mail.gmail.com>
 <2884d6a3-66ca-0897-3284-32dea9db1bc4@lucidity.plus.com>
 <CAPTjJmpazWZcs21FQw-d3E7HaZQd9=XjPUJFBy6BnuojkkjUtA@mail.gmail.com>
 <8c339250-848e-5155-fa5d-c826d81b9da1@lucidity.plus.com>
 <CAPTjJmogyFm0iNegO1xkJdKC+nRTHXyrpZxtUm-2xbM4dTnnMg@mail.gmail.com>
 <mailman.163.1482102287.2322.python-list@python.org>
Message-ID: <8949828.0AQdONaE2F@PointedEars.de>

Chris Angelico wrote:

> On Mon, Dec 19, 2016 at 9:52 AM, Erik <python at lucidity.plus.com> wrote:
>> 1) Method call:
>>    "obj.foo(1, 2, 3)" is syntactic sugar for "obj.foo(obj, 1, 2, 3)".
> 
> And the bit you have to be REALLY careful of when working with both
> Python and JS is that you have to have "obj.foo(...)" as a single
> expression. Consider:
> 
> # Python
> lst = []
> func = lst.append
> func(1); func(2); func(3)
> # lst now holds [1,2,3]
> 
> // JavaScript
> var arr = []
> var func = arr.push.bind(arr) // IMPORTANT
> func(1); func(2); func(3)
> // arr now contains [1,2,3]
> 
> If you just take "arr.push", you don't get a bound method. You get an
> unbound method. You have to explicitly bind that back to the array
> object, or its "this" pointer won't be set.

?this? is not a pointer, it is a built-in variable (adj.) *value*.  There 
are no (high-level) pointers in implementations of ECMAScript (there may be 
in the language used to *build* the *implementation*, e.g. C++).  Instead, 
there are reference values referring to objects.

With func(?), ?this? would be set to a reference to the global object in 
non-strict mode, and to ?undefined? in strict mode.

But the question here is: Why would one insist to declare a (global) 
variable, and assign to it a reference to the Array.prototype.push() method 
instead of calling the ?push? method directly, accessed through the 
prototype chain?  *This* certainly is _not_ how you should do this in 
ECMAScript implementations, but instead:

  /* arr now contains [1,2,3] */
  arr.push(1, 2, 3);

Because ?this? is *always* set to a reference to the calling (or 
constructed) object, if any (if there is no explicit object, it depends on 
the execution mode; see above).

At best, yours is a bad example following a misconception.
 
> This is so important (eg with callback functions) that JS introduced a
> new type of function to help deal with the problem...

Or because "JS" is ?the world?s most misunderstood programming language? 
(Crockford, and I agree with him on *that*), as ISTM you just demonstrated.

There is no ?new type of function?.  Using Function.prototype.bind() this 
way (it is not available in implementations of ECMAScript Ed. 4 and earlier) 
is functionally equivalent to

  func.call(arr, ?, ?);

aso.  It is syntactic sugar so that you can write

  var func2 = func.bind(this);
  func2(?);

and do not have to write, e.g.,

  var that = this;
  var func2 = function () {
    func.call(that, ?, ?);
  };
  func2(?);

but it is not necessarily more efficient than a simple call of 
Function.prototype.call() or Function.prototype.apply().

-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From metal.suomi at gmail.com  Mon Dec 19 08:20:48 2016
From: metal.suomi at gmail.com (metal.suomi at gmail.com)
Date: Mon, 19 Dec 2016 05:20:48 -0800 (PST)
Subject: just started
In-Reply-To: <mailman.120.1481864161.2322.python-list@python.org>
References: <7e1190f3-e5be-4a39-ad68-35abb9832653@googlegroups.com>
 <CAPTjJmqumgxdPunBnQkL6C93i9-ZOAqDB-gOMyJ_Oqqjvcd7Yg@mail.gmail.com>
 <mailman.120.1481864161.2322.python-list@python.org>
Message-ID: <b7b8beab-86ae-42b3-93bf-e005bb144ee2@googlegroups.com>

> In theory, "pip3" will install into the default "python3", whichever
> that is. However, in practice, it's entirely possible that it installs
> into a very different Python from the one you're expecting. The most
> reliable form is the latter; whatever command you use to start Python,
> add "-m pip" to it, and you know you're talking to that same
> installation.

thanks! very clarifying. One more question though: after installing python3.5 I only have pip3 and pip3.5 from the command line. However, I issued the command 

python3.5 -m pip install matplotlib

and it all went like a charm. At the end of the install, it said "You are using pip version 8.1.1, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command"

However, the command 
pip install --upgrade pip' command
doesn't work. So, it seems like pip is not installed. which pip3.5 tells me
/Library/Frameworks/Python.framework/Versions/3.5/bin/pip3.5

so I'm confused. I have only pip3.5 and the command python3.5 -m pip install .... automatically translates into pip3.5 ? 


From ben.usenet at bsb.me.uk  Mon Dec 19 08:48:01 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Mon, 19 Dec 2016 13:48:01 +0000
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
 <o38ciu$9t4$1@dont-email.me>
Message-ID: <87poko3u9a.fsf@bsb.me.uk>

BartC <bc at freeuk.com> writes:

> On 19/12/2016 01:10, Ben Bacarisse wrote:
>> BartC <bc at freeuk.com> writes:
>>
>>> On 18/12/2016 10:59, Paul G?tze wrote:
>>>> there is a nice short article by E. W. Dijkstra about why it makes sense
>>>> to start numbering at zero (and exclude the upper given bound) while
>>>> slicing a list. Might give a bit of additional understanding.
>>>>
>>>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
>>>
>>> (This from somebody who apparently can't use a typewriter?!)
>>>
>>> I don't know if the arguments there are that convincing. Both lower
>>> bounds of 0 and 1 are useful; some languages will use 0, some 1, and
>>> some can have any lower bound.
>>>
>>> But a strong argument for using 1 is that in real life things are
>>> usually counted from 1 (and measured from 0).
>>
>> The index of an element is a measure, not a count.
>
> You need to take your C hat off, I think.

It's a computing hat.  Indexes are best seen as offsets (i.e. as a
measured distances from some origin or base).  It's a model that grew
out of machine addressing and assembler address modes many, many decades
ago -- long before C.  C, being a low-level language, obviously borrowed
it, but pretty much all the well-thought out high-level languages have
seen the value in it too, though I'd be interested in hearing about
counter examples.

The main issue -- of using a half open interval for a range -- is
probably less widely agreed upon, though I think it should be.  EWD is
correct about this (as about so many things).

-- 
Ben.


From skybuck2000 at hotmail.com  Mon Dec 19 08:57:59 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Mon, 19 Dec 2016 05:57:59 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <4c5af126-ab13-441d-bc60-212ad89c5d98@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
 <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
 <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>
 <4c5af126-ab13-441d-bc60-212ad89c5d98@googlegroups.com>
Message-ID: <deff0f2e-fae1-454c-8823-ec4612f8f1d7@googlegroups.com>

Hmmm I see that I have made an old mistake of mine ;)

2x2x2x2 is not 8, it's deceptive... it looks like 4x2 = 8 but nope ! :)

This is 2x2=4x2=8x2=16.

and then the next 4 = 16x2 = 32x2 = 64x2 = 128 x 2=256

so it's roughly 24^4 x 256 = 84934656



From jussi.piitulainen at helsinki.fi  Mon Dec 19 09:06:40 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Mon, 19 Dec 2016 16:06:40 +0200
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
 <o38ciu$9t4$1@dont-email.me> <87poko3u9a.fsf@bsb.me.uk>
Message-ID: <lf5a8bsm2rz.fsf@ling.helsinki.fi>

Ben Bacarisse writes:

> BartC writes:
>
>> You need to take your C hat off, I think.
>
> It's a computing hat.  Indexes are best seen as offsets (i.e. as a
> measured distances from some origin or base).  It's a model that grew
> out of machine addressing and assembler address modes many, many
> decades ago -- long before C.  C, being a low-level language,
> obviously borrowed it, but pretty much all the well-thought out
> high-level languages have seen the value in it too, though I'd be
> interested in hearing about counter examples.

Julia, at version 0.5 of the language, is a major counter-example:
1-based, closed ranges. I think they have been much influenced by the
mathematical practice in linear algebra, possibly through another
computing language.

I think there's some work going on to allow other starting points, or at
least 0. Not sure about half-open ranges.

> The main issue -- of using a half open interval for a range -- is
> probably less widely agreed upon, though I think it should be.  EWD is
> correct about this (as about so many things).

Agreed.

(I even use pen and paper, though I don't always remember what I wrote.)


From skybuck2000 at hotmail.com  Mon Dec 19 09:07:28 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Mon, 19 Dec 2016 06:07:28 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <deff0f2e-fae1-454c-8823-ec4612f8f1d7@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
 <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
 <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>
 <4c5af126-ab13-441d-bc60-212ad89c5d98@googlegroups.com>
 <deff0f2e-fae1-454c-8823-ec4612f8f1d7@googlegroups.com>
Message-ID: <21f47955-5cc5-4809-b9f7-b54351acda26@googlegroups.com>

Hmmm now I think the lookup table cannot work... at least not for the dynamic one... where health is subtracted...

The ships have a certain health... and not just an alive/dead status...

The way they and the enemy attack each other will probably influence the outcome of battle... and for the next round ships will have different health status... so just dead/alive is not enough...

Also pre-computing the table for all 4 or more rounds seems not possible because this again depends on what the enemy was doing...

So I don't think there is a way around the full 24^8 calculations...

Hmmm to bad...


From m at funkyhat.org  Mon Dec 19 10:00:00 2016
From: m at funkyhat.org (Matt Wheeler)
Date: Mon, 19 Dec 2016 15:00:00 +0000
Subject: just started
In-Reply-To: <b7b8beab-86ae-42b3-93bf-e005bb144ee2@googlegroups.com>
References: <7e1190f3-e5be-4a39-ad68-35abb9832653@googlegroups.com>
 <CAPTjJmqumgxdPunBnQkL6C93i9-ZOAqDB-gOMyJ_Oqqjvcd7Yg@mail.gmail.com>
 <mailman.120.1481864161.2322.python-list@python.org>
 <b7b8beab-86ae-42b3-93bf-e005bb144ee2@googlegroups.com>
Message-ID: <CAG93HwFV3PQY6hX6CL=sR1_-yo4ijDthNvaaSOv4YxbWtuta5A@mail.gmail.com>

On Mon, 19 Dec 2016 at 13:20 <metal.suomi at gmail.com> wrote:

> thanks! very clarifying. One more question though: after installing
> python3.5 I only have pip3 and pip3.5 from the command line. However, I
> issued the command
>
> python3.5 -m pip install matplotlib
>
> and it all went like a charm. At the end of the install, it said "You are
> using pip version 8.1.1, however version 9.0.1 is available. You should
> consider upgrading via the 'pip install --upgrade pip' command"
>

I think this is a minor bug in pip. It should probably suggest running pip
in the same way that it was called.


> However, the command
> pip install --upgrade pip' command
> doesn't work. So, it seems like pip is not installed. which pip3.5 tells me
> /Library/Frameworks/Python.framework/Versions/3.5/bin/pip3.5
>
> so I'm confused. I have only pip3.5 and the command python3.5 -m pip
> install .... automatically translates into pip3.5 ?
>

Actually it's the other way around. The `pip3.5` (and `pip3`) commands both
map to (effectively) `python3.5 -m pip`. You can see this if you run `cat
$(which pip3)`.
-- 

--
Matt Wheeler
http://funkyh.at


From bc at freeuk.com  Mon Dec 19 10:48:05 2016
From: bc at freeuk.com (BartC)
Date: Mon, 19 Dec 2016 15:48:05 +0000
Subject: python list index - an easy question
In-Reply-To: <87poko3u9a.fsf@bsb.me.uk>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
 <o38ciu$9t4$1@dont-email.me> <87poko3u9a.fsf@bsb.me.uk>
Message-ID: <o38vd3$gmn$1@dont-email.me>

On 19/12/2016 13:48, Ben Bacarisse wrote:
> BartC <bc at freeuk.com> writes:

>> You need to take your C hat off, I think.
>
> It's a computing hat.  Indexes are best seen as offsets (i.e. as a
> measured distances from some origin or base).

A 1-based or N-based index can still be seen as an offset from element 
0, if that is what you want, although it might be an imaginary element.

But if you needed a table of the frequencies of letters A to Z, indexed 
by the ASCII codes of those letters, then with 0-based you will either 
waste a lot of entries (table[0] to table[64]), or need to build extra 
offsets into the code (table[letter-ord('A')]), or have to use a 
clumsier, less efficient table (ordered dict or whatever).

An N-based array can simply have bounds of ord('A') to ord('Z') 
inclusive. And the array would have just 26 elements.

   It's a model that grew
> out of machine addressing and assembler address modes many, many decades
> ago -- long before C.  C, being a low-level language, obviously borrowed
> it, but pretty much all the well-thought out high-level languages have
> seen the value in it too, though I'd be interested in hearing about
> counter examples.

Older languages tend to use 1-based indexing. Newer ones zero, but it's 
possible C has been an influence if that is what designers and 
implementers have been exposed to.

Among the 1-based or N-based languages are Fortran, Algol68 and Ada:

https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(array)

(I'd only known 1- or N-based ones so the languages I've created have 
all been N-based with a default of 1. With exceptions for certain types, 
eg: strings are 1-based only; bit-indexing within machine words is 0-based.)

However, if the base for arrays is fixed at 0 or 1, then 0-based is 
probably more flexible, as you can emulated 1-based indexing by adding 
an extra element and ignoring element 0; it's a little harder the other 
way around!

> The main issue -- of using a half open interval for a range -- is
> probably less widely agreed upon, though I think it should be.  EWD is
> correct about this (as about so many things).

With 0-based lists, then half-open intervals are more suited. Otherwise 
a list of N elements would be indexed by a range of [0:N-1]. While 
1-based is more suited to closed intervals: [1:N].

But then, suppose you also had an alternate syntax for a range, of 
[A^^N], which starts at A and has a length of N elements. Then ranges of 
[1^^N] and [0^^N] are both specified the same way. You don't need to 
worry about open or closed intervals.

(I've seen two arrows ^^ used on road signs to denote the length of an 
impending tunnel rather than the distance to it.)

-- 
Bartc




From brostype at gmail.com  Mon Dec 19 11:11:04 2016
From: brostype at gmail.com (brostype at gmail.com)
Date: Mon, 19 Dec 2016 08:11:04 -0800 (PST)
Subject: how to run github codes on qpython?
Message-ID: <d9294e16-92e6-423e-b5c5-cbda9867b6ac@googlegroups.com>

hello im Android user and i was learning python programming language from a month ago with qpython
and i have a question is it possible to run this github scripts on qpython? 
https://github.com/rmmh/skybot

thnx.


From ohiomcbd at gmail.com  Mon Dec 19 11:14:42 2016
From: ohiomcbd at gmail.com (Jed Mack)
Date: Mon, 19 Dec 2016 11:14:42 -0500
Subject: Problem running Python 3.5.2 on school network PC
In-Reply-To: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
References: <CAMZ3PMUrRb0XdGA7svk=E7r9kZEmKkVcYkpOT0d=kGJtQ1Nkeg@mail.gmail.com>
Message-ID: <CAMZ3PMVjJ_kB0YFYAL9gUT2bLh1xsBGOwifjNqAVtVT0Y+vtNQ@mail.gmail.com>

This problem has been solved.  Thanks to all of you for your suggestions.
Below is a summary I sent to Tim Golden, whose reply led us to the solution
in this case.

Tim,

Thank you for your help  -  your suggestions led us to the problem.  Turns
out that another instance of Python *was* installed, not as part of OEM
software, but by a digital camera (Califone) that is being used in some of
our classrooms.  The suggestion to look for environmental variables was the
key  -  on the PCs where we were having the problem, there was a variable
starting with PYTHON ... which pointed to the camera software.  Once the
camera was uninstalled, Python 3.5.2 worked just fine.

Thanks again!

On Thu, Dec 15, 2016 at 11:11 AM, Jed Mack <ohiomcbd at gmail.com> wrote:

> We are having a problem running Python 3.5.2 on Windows 10 x64 computers,
> which are members of a school network.
>
>
>
> The program seems to install correctly, but when we try to run the program
> it stops and give an error message saying:
>
>
>
> *Fatal Python error: Py_Initialize: unable to load the file system codec*
>
> *Traceback (most recent call last):*
>
> *  File "C:\Program Files\Python35\lib\encodings\__init__.py", line 31,
> in< module>*
>
> *zipimport.ZipImportError: can't find module 'codecs'*
>
>
>
>
>
> On a Windows 7 PCs, on the same network, the program runs with no problems.
>
>
>
> We have no one here who is familiar with Python.  Do you have any
> additional information on this error, and suggestions for fixing it?
>
>
>
> We have a teacher who needs this program on Windows 10 PCs for students to
> use beginning January 3.
>
>
>
> Thanks,
>
> Jed Mack
>
>
>
>


From ben.usenet at bsb.me.uk  Mon Dec 19 15:44:39 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Mon, 19 Dec 2016 20:44:39 +0000
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
 <o38ciu$9t4$1@dont-email.me> <87poko3u9a.fsf@bsb.me.uk>
 <lf5a8bsm2rz.fsf@ling.helsinki.fi>
Message-ID: <8760mf7io8.fsf@bsb.me.uk>

Jussi Piitulainen <jussi.piitulainen at helsinki.fi> writes:

> Ben Bacarisse writes:
>
>> BartC writes:
>>
>>> You need to take your C hat off, I think.
>>
>> It's a computing hat.  Indexes are best seen as offsets (i.e. as a
>> measured distances from some origin or base).  It's a model that grew
>> out of machine addressing and assembler address modes many, many
>> decades ago -- long before C.  C, being a low-level language,
>> obviously borrowed it, but pretty much all the well-thought out
>> high-level languages have seen the value in it too, though I'd be
>> interested in hearing about counter examples.
>
> Julia, at version 0.5 of the language, is a major counter-example:
> 1-based, closed ranges. I think they have been much influenced by the
> mathematical practice in linear algebra, possibly through another
> computing language.

Interesting.  Thanks.

<snip>
-- 
Ben.


From greg.ewing at canterbury.ac.nz  Mon Dec 19 16:43:35 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 20 Dec 2016 10:43:35 +1300
Subject: python list index - an easy question
In-Reply-To: <o38vd3$gmn$1@dont-email.me>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me> <87zijs4tb4.fsf@bsb.me.uk>
 <o38ciu$9t4$1@dont-email.me> <87poko3u9a.fsf@bsb.me.uk>
 <o38vd3$gmn$1@dont-email.me>
Message-ID: <ebr2k9Fr5v6U1@mid.individual.net>

BartC wrote:
> But if you needed a table of the frequencies of letters A to Z...
> An N-based array can simply have bounds of ord('A') to ord('Z') 
> inclusive.

That's fine if your language lets you have arrays with
arbitrary lower bounds.

But if the language only allows a fixed lower bound, and
furthermore insists that the lower bound be 1, then your
indexing expression becomes:

    table[ord(letter) - ord('A') + 1]

If a language is only going to allow me a single lower
bound, I'd rather it be 0, because I can easily shift
that by whatever offset I need. But if it's 1, often
I need to cancel out an unwanted 1 first, leading to code
that's harder to reason about.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Dec 19 16:58:08 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 20 Dec 2016 10:58:08 +1300
Subject: OT - "Soft" ESC key on the new MacBook Pro
In-Reply-To: <mailman.166.1482152256.2322.python-list@python.org>
References: <CANc-5UxU=9F0KAe12OuEhRuHYdWoUvsqGdYEiz4hLEwbnHfnpA@mail.gmail.com>
 <CANc-5UxXPRF3qYKEmnyDdANoN-5GLiRXDrOU6E=R-8N8ZV2UdQ@mail.gmail.com>
 <CANc-5UwNFgS4aFN5qbwOPNdTMfXrEpJrv4b00_ycGmwAAoK81Q@mail.gmail.com>
 <CANc-5UyhtmjE9rXwhMBx=H9tBmYARhfzVjw2O=iKDmyEdCpqSA@mail.gmail.com>
 <CANc-5UyHLP3W8-_D18f8qE-9z0Y-DMbuHRMLpLY10eA52YZMew@mail.gmail.com>
 <CANc-5UxQA=-9Q=1GPh+5epX9E3KbAgd6Ye_63=kB55aB-MHqow@mail.gmail.com>
 <CANc-5UwqKjfxrxS0tvSk6r0kRQbr5kiAyCxJhUT8JPfcySSG2A@mail.gmail.com>
 <CANc-5Uyj9VnJbVGSepVLLHeCgoB9uAOD61Ft4b=bdLTQJ5dE=A@mail.gmail.com>
 <mailman.63.1481678752.2322.python-list@python.org>
 <ebdeh2FhsjbU1@mid.individual.net>
 <CANc-5UxL4H3mnp8Rg-MDDSJ4rovx6=83p1-VF2_xODiUSB=vNw@mail.gmail.com>
 <mailman.76.1481737833.2322.python-list@python.org>
 <ebg4apF76f9U1@mid.individual.net> <o36vge$kr1$2@dont-email.me>
 <ebofe8F7uovU1@mid.individual.net>
 <1482152253.1697031.823531193.25AC26DA@webmail.messagingengine.com>
 <mailman.166.1482152256.2322.python-list@python.org>
Message-ID: <ebr3fiFrcfdU1@mid.individual.net>

Random832 wrote:
> Except for the fact that the actual keyboard that Emacs was originally
> developed for [the Knight Keyboard, and the later Symbolics "Space
> Cadet" Keyboards] had the control key more or less where it is on modern
> PC keyboards [slightly further to the right, so easier to reach with the
> thumb of the same-side hand]

If it's reachable with the thumb, that's actually quite
a big difference, because the thumb is much stronger than
the little finger, therefore better suited for a key that's
so frequently used.

> Meanwhile, users of Unix systems were stuck with terminals
> like the ADM-3A and VT-100, which, inherited from the ASR-33 but one
> must assume the real reason was to save money,

Probably they never anticipated that the control key would
become such a frequently-used key. All the commonly-used
ASCII control characters (CR, LF, TAB etc.) had their own
dedicated keys, so you only needed to use the control key
for characters that were rarely used before things like
Emacs came along.

-- 
Greg


From skybuck2000 at hotmail.com  Mon Dec 19 19:45:13 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Mon, 19 Dec 2016 16:45:13 -0800 (PST)
Subject: Best attack order for groups of numbers trying to destroy each
 other, given a victory chance for number to number attack.
In-Reply-To: <21f47955-5cc5-4809-b9f7-b54351acda26@googlegroups.com>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <e58e0cd9-a4e6-4ac7-8be6-adc4eff1017b@googlegroups.com>
 <401a613b-b784-4a62-adfb-831362676d6c@googlegroups.com>
 <3c5b94c1-e56d-4a95-8365-dafa0980d6e4@googlegroups.com>
 <4c5af126-ab13-441d-bc60-212ad89c5d98@googlegroups.com>
 <deff0f2e-fae1-454c-8823-ec4612f8f1d7@googlegroups.com>
 <21f47955-5cc5-4809-b9f7-b54351acda26@googlegroups.com>
Message-ID: <ca11db2d-2e77-4e77-8ac7-bb2b7248e699@googlegroups.com>

I first explored the possibility of a lookup table to speed up calculations, this possibility would probably give wrong results.

I am interested in calculating all possibilities to see which one is truely best, I am also interested in trying to speed up calculations to be able to handle larger models.

Many other possibilities exist for further exploration:

1. One recursive function to try and split up the work as to recycle some of it.
2. Perhaps multiple recursive functions to include different functionality at different "depths".
3. Other tree like structures to try and safe some computations by recycling results from previous visits across nodes.
4. Manually incrementing indexes (though this would probably requiring storing results in some form or another, perhaps a stack based approach might work... pushing/popping as index returns to a certain one, perhaps a stack per index might work or something.
5. Detecting when index changes and then reloading previous computations.

I still believe that it should be possible somehow to save some computations.

Perhaps I am fooling myself ! Not sure yet ! ;) :)

6. Healths will need to be reset I think, however another possibility is copy&pasting healths to new nodes to compute them further.

Sort of like a propagation algorithm... spreading already calculated results to other nodes.

One problem with all that node thinking is that the system will run out of memory, which should be fine, memory can be pre-allocated to speed stuff up, one it runs out, it will have to do full calculations.

I wrote about this before but cancelled posting cause it was kinda messy... but now it makes a bit more sense, now this text a bit more clear.

7. Maybe cuda or some other SIMD solution. Still not sure if CUDA can detect that some instructions/data are being calculated... hmmm would guess not, not sure though.

It's kinda fun trying out all these possibilities to see which ones work... though I am also becoming a bit curious to what the actual outcome would be.

For now my heuristic for a "good enough" solution would be "object attack object with best victory chance".

This seems a bit too easy/cheesy heuristic and I want to know what is really the best way... is there perhaps a team strategy/combined attack effort that might work better ? ;)

I would also like to be able to change chances perhaps in feature or expand model a little bit so also speeding it up would be nice.

For now multi threading is doable... and the calculate while idle seems nice...

Though I am also kinda curious about cuda... these last ideas are probably easiest to implement and will give most statisfactory.

One problem with that is though, once I have the problem solved/calculated I might not be interested in trying out the harder possibilities.

So maybe it's best to try the hardest first... maybe I will learn something amuzing and even something usefull from it.

More failures perhaps, but in the end, success will come ! LOL.


From steve+python at pearwood.info  Mon Dec 19 19:49:19 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 20 Dec 2016 11:49:19 +1100
Subject: python list index - an easy question
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
Message-ID: <58588011$0$1597$c3e8da3$5496439d@news.astraweb.com>

On Mon, 19 Dec 2016 03:21 am, BartC wrote:

> On 18/12/2016 10:59, Paul G?tze wrote:
>> Hi John,
>>
>> there is a nice short article by E. W. Dijkstra about why it makes sense
>> to start numbering at zero (and exclude the upper given bound) while
>> slicing a list. Might give a bit of additional understanding.
>>
>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
> 
> (This from somebody who apparently can't use a typewriter?!)

Oh Bart don't be so hard on yourself, I don't think it matters whether or
not you can use a typewriter, your comments would be as equally interesting
whether you used a typewriter, spoke them allowed and had them transcribed
by voice recognition software, or wrote them out by hand.

(Handwriting is a lost skill in this day and age of people poking at their
JesusPhones with their thumb, typing out txt sp3k 1 ltr at a time.)


> I don't know if the arguments there are that convincing. Both lower
> bounds of 0 and 1 are useful; 

Dijkstra doesn't deny that other bounds can be useful.

The unstated assumption in his argument is you can only pick one system for
indexing into arrays. But of course that's not strictly correct --
different languages can and do choose different schemes, or even choose
different schemes for each language.


> some languages will use 0, some 1, and 
> some can have any lower bound.
> 
> But a strong argument for using 1 is that in real life things are
> usually counted from 1 (and measured from 0).

I'm sure that when European merchants first learned of this
new-fangled "zero" from their Arabic and Indian colleagues in the Middle
Ages, they probably had a quite similar reaction.

After all, didn't the Greeks themselves claim that the smallest number was
two? Zero is clearly not a number, it is the absence of number, and 1 is
not a number, it is unity.

http://math.furman.edu/~mwoodard/fuejum/content/2012/paper1_2012.pdf

You do make a point that even children (at least those about the age of
three or four) can understand 1-based indexing, but I don't think it is a
*critical* point. Most programmers have learned a bit more than the average
four year old.

If you can understand threads, OOP design patterns, binary search or C
pointers, counting indexes 0, 1, 2, 3, ... should hold no fears for you.


> So if you wanted a simple list giving the titles of the chapters in a
> book or on a DVD, on the colour of the front doors for each house in a
> street, usually you wouldn't be able to use element 0.
> 
> As for slice notation, I tend to informally use (not for any particulr
> language) A..B for an inclusive range, and A:N for a range of length N
> starting from A.

An interesting thought... I like the idea of being able to specify slices by
index or by length.


 
> In Python you can also have a third operand for a range, A:B:C, which
> can mean that B is not necessarily one past the last in the range, and
> that the A <= i < B condition in that paper is no longer quite true.

You cannot use a simple comparison like A <= i < B to represent a range with
a step-size not equal to 1. How would you specify the sequence:

    2, 4, 6, 8, 10, 12, 14

as a single expression with ONLY an upper and lower bound?

    2 <= i <= 14

clearly doesn't work, and nor does any alternative.

Dijkstra doesn't discuss range() with non-unit step sizes.


> In fact, A:B:-1 corresponds to A >= i > B, which I think is the same as
> condition (b) in the paper (but backwards), rather (a) which is favoured.

I have no way of knowing what it corresponds to in your head, but in Python,
a slice A:B:-1 corresponds to the elements at indices:

    A, A-1, A-2, A-3, ..., B+1

*in that order*.

py> '0123456789'[7:2:-1]
'76543'

Since Dijkstra doesn't discuss non-unit step sizes, it is unclear what he
would say about counting backwards.


> Another little anomaly in Python is that when negative indices are used,
> it suddenly switches to 1-based indexing! Or least, when -index is
> considered:
> 
>    x = [-4,-3,-2,-1]
> 
>    print x[-1]       # -1  Notice the correspondence here...
>    print x[-2]       # -2

Congratulations, you've discovered that 4-1 equals 3. Well done!

>    x = [1, 2, 3, 4]
> 
>    print x[1]        # 2   ...and the lack of it here
>    print x[2]        # 3

That's silly. You can get whichever correspondence you like by choosing
values which do or don't match the indices used by Python:

    x = [-3, -2, -1, 0]
    print x[-1]  # Notice the lack of correspondence

    x = [0, 1, 2, 3]
    print x[1]  # Notice the correspondence here

    x = [6, 7, 8, 9]
    print x[1]  # Notice the lack of correspondence here

What's your point?


I infer that you're having difficulty with Python slicing and indexing
because you have a mental model where indexes label the items themselves
(best viewed in a fixed width font):

+---+---+---+---+---+---+
| P | y | t | h | o | n |
+---+---+---+---+---+---+
..0...1...2...3...4...5        # Indexes
.-6..-5..-4..-3..-2..-1        # 6-1 = 5, 6-2 = 4 etc.


That makes 'Python'[1] easy to visualise (its obviously just 'y') but
slicing becomes more complex:

    'Python'[1:4]
    => *include* index 1, *exclude* index 4
    => 'yth'

But a better model is to consider the *boundaries* labelled:

+---+---+---+---+---+---+
| P | y | t | h | o | n |
+---+---+---+---+---+---+
0...1...2...3...4...5...6        # Indexes
-6..-5..-4..-3..-2..-1


A simple index is a bit more complex (think of it as the element immediately
following the boundary, hence 'Python'[1] is still 'y') but slicing now
becomes simple. Visualise slicing along the boundaries (that's where the
name comes from!) and now it is obvious that the start index is included
and the stop index is excluded/

+---+---+---+---+---+---+
| P | y | t | h | o | n |
+---+---+---+---+---+---+
....^...........^
cut here    and here


There's no mental model that makes slicing with non-zero step sizes easy,
*especially* if the step size is a non-unit negative number (say, -3). For
those you really have to simulate the process of actually iterating over
the indexes one by one.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ozpeterballard at gmail.com  Tue Dec 20 00:08:35 2016
From: ozpeterballard at gmail.com (ozpeterballard at gmail.com)
Date: Mon, 19 Dec 2016 21:08:35 -0800 (PST)
Subject: Can't see numpy etc after upgrading Python on Ubuntu
Message-ID: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>

Hi all. I hope this is the right place. It seems more like a Python problem than an Ubuntu problem.

I use Python on Ubuntu 12.04. Recently I upgraded Python from 2.7.3 to 2.7.12. I did the upgrade via the source, i.e. download (from https://www.python.org/downloads/release/python-2712/ ), tar, ./config, sudo make install.

After the upgrade, python could no longer see the 3rd party modules I use (numpy, scipy, matplotlib, cython). I can point PYTHONPATH to the cython files, but I can't work out how to include numpy, scipy and matplotlib.

It's no good asking Ubuntu to upgrade, because apt-get tells me that numpy, scipy and matplotlib are all up to date.

So what do I do so I can see these packages again?

p.s. results of a few commands:

$ which python
/usr/local/bin/python

$ find /usr -name 'numpy*'
/usr/share/pyshared/numpy-1.6.1.egg-info
/usr/share/pyshared/scipy/weave/numpy_scalar_spec.py
/usr/share/pyshared/numpy
/usr/share/pyshared/numpy/core/include/numpy
/usr/share/pyshared/numpy/core/include/numpy/numpyconfig.h
/usr/share/pyshared/numpy/numarray/include/numpy
/usr/share/pyshared/numpy/testing/numpytest.py
/usr/share/pyshared/numpy/distutils/numpy_distribution.py
/usr/share/numpy
/usr/lib/python2.7/dist-packages/numpy-1.6.1.egg-info
/usr/lib/python2.7/dist-packages/scipy/weave/numpy_scalar_spec.py
/usr/lib/python2.7/dist-packages/scipy/weave/numpy_scalar_spec.pyc
/usr/lib/python2.7/dist-packages/numpy
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/numpyconfig.h
/usr/lib/python2.7/dist-packages/numpy/numarray/include/numpy
/usr/lib/python2.7/dist-packages/numpy/testing/numpytest.py
/usr/lib/python2.7/dist-packages/numpy/testing/numpytest.pyc
/usr/lib/python2.7/dist-packages/numpy/distutils/numpy_distribution.pyc
/usr/lib/python2.7/dist-packages/numpy/distutils/numpy_distribution.py
/usr/lib/pyshared/python2.7/numpy
/usr/local/lib/python2.7/dist-packages/Cython/Includes/numpy
/usr/include/python2.7/numpy
/usr/include/python2.7_d/numpy
/usr/include/numpy

... and fairly similar results for scipy and matplotlib.

My bash PYTHONPATH is:

PYTHONPATH=:/home/peter/dvd/software/python:/home/peter/Cython-0.25.2

My sys.path in python is:

>>> sys.path
['', '/home/peter', '/home/peter/dvd/software/python', '/home/peter/Cython-0.25.2', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages']


From rosuav at gmail.com  Tue Dec 20 00:36:15 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 20 Dec 2016 16:36:15 +1100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
Message-ID: <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>

On Tue, Dec 20, 2016 at 4:08 PM,  <ozpeterballard at gmail.com> wrote:
> I use Python on Ubuntu 12.04. Recently I upgraded Python from 2.7.3 to 2.7.12. I did the upgrade via the source, i.e. download (from https://www.python.org/downloads/release/python-2712/ ), tar, ./config, sudo make install.
>
> After the upgrade, python could no longer see the 3rd party modules I use (numpy, scipy, matplotlib, cython). I can point PYTHONPATH to the cython files, but I can't work out how to include numpy, scipy and matplotlib.
>
> It's no good asking Ubuntu to upgrade, because apt-get tells me that numpy, scipy and matplotlib are all up to date.
>
> So what do I do so I can see these packages again?

First off, you're running a still-in-support old version of Ubuntu, so
check whether you can upgrade Python using apt-get. Check not just the
version number, but any features you need. I don't know whether Ubuntu
do this, but Red Hat often backport bug fixes and security patches
onto older builds of Python; what you get is named according to the
oldest component in it, but might well have a number of newer
features.

But once you decide to build Python from source, you have to choose
between overwriting your existing Python (probably in /usr/bin) or
creating a new one (in /usr/local/bin). By default, untarring,
configuring without args, building, and installing, will give you
something in /usr/local/bin - which most likely means you're leaving
behind all the Python modules installed with apt-get. You also won't
touch anything that uses libpython, or anything like that. This is the
safe option. So, how do you get numpy etc into your new Python? Most
likely, using pip. Try this:

sudo apt-get build-dep python-numpy
sudo python -m pip install numpy

That'll build numpy from source. (I may have the package name wrong;
that's what it's called on my Debian, but it might be different on
Ubuntu.) It'll take a while, so if you're on a system with a slow CPU
and/or not much memory, be patient. But that should get numpy going
for you.

ChrisA


From ozpeterballard at gmail.com  Tue Dec 20 01:19:44 2016
From: ozpeterballard at gmail.com (ozpeterballard at gmail.com)
Date: Mon, 19 Dec 2016 22:19:44 -0800 (PST)
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <mailman.171.1482212178.2322.python-list@python.org>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
Message-ID: <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>

Thanks Chris for replying, but it didn't work. The upgrade happened, but still python can't see numpy! So it seems to be a path problem. The numpy (and scipy and matplotlib) files are there, so surely in principle it's a simple matter of pointing my python path at them?

Any ideas how?

(I should add, the reason I upgraded python from source in the first place is that "apt-get" claimed python was up to date).

p.s. Here's an abridged log:

peter at sirboris:~$ which python
/usr/local/bin/python

peter at sirboris:~$ pip
The program 'pip' is currently not installed.  You can install it by typing:
sudo apt-get install python-pip

peter at sirboris:~$ sudo apt-get install python-pip
[lots of output, pip installed]

peter at sirboris:~$ which pip
/usr/bin/pip

peter at sirboris:~$ sudo pip install numpy
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python2.7/dist-packages
Cleaning up...

peter at sirboris:~$ sudo pip install numpy --upgrade
[lots and lots of output, successful install]

peter at sirboris:~$ python
Python 2.7.12 (default, Dec 11 2016, 22:16:38) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> import sys
>>> sys.path
['', '/home/peter', '/home/peter/dvd/software/python', '/home/peter/Cython-0.25.2', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages']

# [Then I thought, maybe "pip" and "python -m pip" are different, so...]

peter at sirboris:~$ sudo python -m pip install numpy
/usr/local/bin/python: No module named pip


From rosuav at gmail.com  Tue Dec 20 02:45:32 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 20 Dec 2016 18:45:32 +1100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
 <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
Message-ID: <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>

On Tue, Dec 20, 2016 at 5:19 PM,  <ozpeterballard at gmail.com> wrote:
> Thanks Chris for replying, but it didn't work. The upgrade happened, but still python can't see numpy! So it seems to be a path problem. The numpy (and scipy and matplotlib) files are there, so surely in principle it's a simple matter of pointing my python path at them?
>
> Any ideas how?

You just installed pip into your /usr/bin/python, not
/usr/local/bin/python. You'll need to get pip in that Python and use
that. I'm not sure the best way to install pip into a 2.7 - someone
else on this list can advise.

ChrisA


From vincent.vande.vyvre at telenet.be  Tue Dec 20 03:36:20 2016
From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre)
Date: Tue, 20 Dec 2016 09:36:20 +0100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
 <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
 <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>
Message-ID: <a025cebe-9d9d-54d4-fd5a-8cab92c095e8@telenet.be>

Le 20/12/16 ? 08:45, Chris Angelico a ?crit :
> On Tue, Dec 20, 2016 at 5:19 PM,  <ozpeterballard at gmail.com> wrote:
>> Thanks Chris for replying, but it didn't work. The upgrade happened, but still python can't see numpy! So it seems to be a path problem. The numpy (and scipy and matplotlib) files are there, so surely in principle it's a simple matter of pointing my python path at them?
>>
>> Any ideas how?
> You just installed pip into your /usr/bin/python, not
> /usr/local/bin/python. You'll need to get pip in that Python and use
> that. I'm not sure the best way to install pip into a 2.7 - someone
> else on this list can advise.
>
> ChrisA

It seems you have shadowed your default python2 with the new one. A very 
bad idea.

You can have two python2 but you can't replace the default version.

The default is in /usr/lib and the new is in /usr/locale/lib

The third party libs are into the default version. For this reason 
there's probably some applications that are broken. (apport, bzr, 
synaptic?, ...)

The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, 
restore it.

One solution for use the new version is to create the necessary symbolic 
links into /usr/locale/lib/python2.7/dist-packages which point to 
/usr/lib/python2.7/dist-packages numpy, scipy and others.

In fact, this is the same think when you create a virtual environment, 
the libs are not copied but linked.


VVV



From rosuav at gmail.com  Tue Dec 20 04:13:44 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 20 Dec 2016 20:13:44 +1100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <a025cebe-9d9d-54d4-fd5a-8cab92c095e8@telenet.be>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
 <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
 <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>
 <a025cebe-9d9d-54d4-fd5a-8cab92c095e8@telenet.be>
Message-ID: <CAPTjJmoC57KmnV0j2Eoc=r0=zLHeRC2yrjp4DHRjQBdgyhjeKg@mail.gmail.com>

On Tue, Dec 20, 2016 at 7:36 PM, Vincent Vande Vyvre
<vincent.vande.vyvre at telenet.be> wrote:
> It seems you have shadowed your default python2 with the new one. A very bad
> idea.

Only because /usr/local/bin is ahead of /usr/bin in PATH.

> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, restore
> it.

It does. When he installs pip via apt, it manages his original 2.7.3.
Anything that explicitly shebangs to /usr/bin/python will be
unaffected by this.

ChrisA


From vincent.vande.vyvre at telenet.be  Tue Dec 20 04:28:54 2016
From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre)
Date: Tue, 20 Dec 2016 10:28:54 +0100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <CAPTjJmoC57KmnV0j2Eoc=r0=zLHeRC2yrjp4DHRjQBdgyhjeKg@mail.gmail.com>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
 <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
 <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>
 <a025cebe-9d9d-54d4-fd5a-8cab92c095e8@telenet.be>
 <CAPTjJmoC57KmnV0j2Eoc=r0=zLHeRC2yrjp4DHRjQBdgyhjeKg@mail.gmail.com>
Message-ID: <9636fb5a-3445-8167-7248-cc72986b2a30@telenet.be>

Le 20/12/16 ? 10:13, Chris Angelico a ?crit :
> On Tue, Dec 20, 2016 at 7:36 PM, Vincent Vande Vyvre
> <vincent.vande.vyvre at telenet.be> wrote:
>> It seems you have shadowed your default python2 with the new one. A very bad
>> idea.
> Only because /usr/local/bin is ahead of /usr/bin in PATH.
That's the problem.
>
>> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, restore
>> it.
> It does. When he installs pip via apt, it manages his original 2.7.3.
> Anything that explicitly shebangs to /usr/bin/python will be
> unaffected by this.
>
> ChrisA

Not sure, when the OP enter python into a terminal he have:

peter at sirboris:~$ python
Python 2.7.12 (default, Dec 11 2016, 22:16:38)

So, python point now to the new one. No? VVV



From rosuav at gmail.com  Tue Dec 20 04:33:27 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 20 Dec 2016 20:33:27 +1100
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <9636fb5a-3445-8167-7248-cc72986b2a30@telenet.be>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
 <CAPTjJmpqKhZi3aL2mD_6YvunbkW4ukU6su14EXL4xi8yNO_gXg@mail.gmail.com>
 <mailman.171.1482212178.2322.python-list@python.org>
 <4974b971-479c-45c3-94e3-b47498cfea30@googlegroups.com>
 <CAPTjJmoWvbr--Rm+91Q5A+DvOqvhj9K-djcRFuAq+o9m4wRpUw@mail.gmail.com>
 <a025cebe-9d9d-54d4-fd5a-8cab92c095e8@telenet.be>
 <CAPTjJmoC57KmnV0j2Eoc=r0=zLHeRC2yrjp4DHRjQBdgyhjeKg@mail.gmail.com>
 <9636fb5a-3445-8167-7248-cc72986b2a30@telenet.be>
Message-ID: <CAPTjJmrNv_8BUFhCoY99h1XZf86Yn7p-ydG9t2W4sDXUVUUxPw@mail.gmail.com>

On Tue, Dec 20, 2016 at 8:28 PM, Vincent Vande Vyvre
<vincent.vande.vyvre at telenet.be> wrote:
>>> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not,
>>> restore
>>> it.
>>
>> It does. When he installs pip via apt, it manages his original 2.7.3.
>> Anything that explicitly shebangs to /usr/bin/python will be
>> unaffected by this.
>>
>> ChrisA
>
>
> Not sure, when the OP enter python into a terminal he have:
>
> peter at sirboris:~$ python
> Python 2.7.12 (default, Dec 11 2016, 22:16:38)
>
> So, python point now to the new one. No? VVV

So when you type 'python' at the command line, it searches $PATH. But
you don't need to worry about system scripts; they'll use an explicit
"#!/usr/bin/python", which means they'll keep running 2.7.3.

(Unless, of course, the goal of upgrading was actually to change the
system Python, in which case you want to do things _very_
differently.)

ChrisA


From bc at freeuk.com  Tue Dec 20 06:45:07 2016
From: bc at freeuk.com (BartC)
Date: Tue, 20 Dec 2016 11:45:07 +0000
Subject: python list index - an easy question
In-Reply-To: <58588011$0$1597$c3e8da3$5496439d@news.astraweb.com>
References: <2cef3e05-e3d6-47b1-b9fd-b6b85e0d5d66@googlegroups.com>
 <fa9c6866-c62f-921b-1f17-c68956737e62@gmail.com>
 <mailman.156.1482070912.2322.python-list@python.org>
 <o36cvf$duq$1@dont-email.me>
 <58588011$0$1597$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o3b5hh$nrg$1@dont-email.me>

On 20/12/2016 00:49, Steve D'Aprano wrote:
> On Mon, 19 Dec 2016 03:21 am, BartC wrote:
>
>> On 18/12/2016 10:59, Paul G?tze wrote:
>>> Hi John,
>>>
>>> there is a nice short article by E. W. Dijkstra about why it makes sense
>>> to start numbering at zero (and exclude the upper given bound) while
>>> slicing a list. Might give a bit of additional understanding.
>>>
>>> http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF

> If you can understand threads, OOP design patterns, binary search or C
> pointers, counting indexes 0, 1, 2, 3, ... should hold no fears for you.

Who's talking about me? In the area of scripting languages that could be 
used by people with a wide range of expertise, I would have expected 
1-based to be more common.

>> In Python you can also have a third operand for a range, A:B:C, which
>> can mean that B is not necessarily one past the last in the range, and
>> that the A <= i < B condition in that paper is no longer quite true.
>
> You cannot use a simple comparison like A <= i < B to represent a range with
> a step-size not equal to 1. How would you specify the sequence:
>
>     2, 4, 6, 8, 10, 12, 14
>
> as a single expression with ONLY an upper and lower bound?

>     2 <= i <= 14
>
> clearly doesn't work, and nor does any alternative.

I didn't mean that the expression specifies the range, but that an value 
in the range satisfies the expression.

>> In fact, A:B:-1 corresponds to A >= i > B, which I think is the same as
>> condition (b) in the paper (but backwards), rather (a) which is favoured.
>
> I have no way of knowing what it corresponds to in your head, but in Python,
> a slice A:B:-1 corresponds to the elements at indices:

I got the idea for a minute that, since A:B:C specifies a set of 
indices, and that range(A,B,C) specifies the same set of values, that 
they are interchangeable. But of course you can't use x[range(A,B,C)] 
and you can't do 'for i in A:B:C'. [In my, ahem, own language, you can 
just that...]

>     A, A-1, A-2, A-3, ..., B+1
>
> *in that order*.
>
> py> '0123456789'[7:2:-1]
> '76543'

When A is 7, B is 2 and C is -1, then

for i in range(-1000,1000):
     if A >=i > B:
         print (i)

displays only the numbers 3,4,5,6,7; the same elements you get, and they 
satisfy A >= i > B as I said.

>> Another little anomaly in Python is that when negative indices are used,
>> it suddenly switches to 1-based indexing! Or least, when -index is
>> considered:

> That's silly. You can get whichever correspondence you like by choosing
> values which do or don't match the indices used by Python:
>
>     x = [-3, -2, -1, 0]
>     print x[-1]  # Notice the lack of correspondence
>
>     x = [0, 1, 2, 3]
>     print x[1]  # Notice the correspondence here
>
>     x = [6, 7, 8, 9]
>     print x[1]  # Notice the lack of correspondence here

> What's your point?

That point is that indices go 0,1,2,3,... when indexed from the start, 
and -1,-2,-3,... when indexed from the end.

That means that the THIRD item from the start has index [2], but the 
THIRD item from the end has index [-3].

(So if you reversed a list x, and wanted the same x[i] element you had 
before it was reversed, it now has to be x[-(i+1)]. 1-based, the 
elements would be x[i] and x[-i], a little more symmetric. But with 
N-based, you wouldn't be able to index from the end at all.)

> +---+---+---+---+---+---+
> | P | y | t | h | o | n |
> +---+---+---+---+---+---+
> ....^...........^
> cut here    and here

That's the fence/fencepost thing I mentioned elsewhere. It comes up also 
in graphics: if these boxes represent pixels rather than elements, and a 
function draws a line from pixel 1 to pixel 4 or fills then in, then 
should pixel 4 be filled in or not?

But if you think of these values as continuous measures from the left 
edge, rather than as discrete units, and denote them as 1.0 to 4.0, then 
it becomes more obvious. The 3 pixels between 1.0 and 4.0 are filled in.

-- 
bartc


From flebber.crue at gmail.com  Tue Dec 20 06:53:42 2016
From: flebber.crue at gmail.com (Sayth Renshaw)
Date: Tue, 20 Dec 2016 03:53:42 -0800 (PST)
Subject: windows utf8 & lxml
Message-ID: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>

Hi 

I have been trying to get a script to work on windows that works on mint. The key blocker has been utf8 errors, most of which I have solved.

Now however the last error I am trying to overcome, the solution appears to be to use the .decode('windows-1252') to correct an ascii error.

I am using lxml to read my content and decode is not supported are there any known ways to read with lxml and fix unicode faults?

The key part of my script is 

        for content in roots:
            utf8_parser = etree.XMLParser(encoding='utf-8')
            fix_ascii = utf8_parser.decode('windows-1252')
            mytree = etree.fromstring(
                content.read().encode('utf-8'), parser=fix_ascii)

Without the added .decode my code looks like

        for content in roots:
            utf8_parser = etree.XMLParser(encoding='utf-8')
            mytree = etree.fromstring(
                content.read().encode('utf-8'), parser=utf8_parser)

However doing it in such a fashion returns this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
Which I found this SO for http://stackoverflow.com/a/29217546/461887 but cannot seem to implement with lxml.

Ideas?

Sayth


From ozpeterballard at gmail.com  Tue Dec 20 07:11:28 2016
From: ozpeterballard at gmail.com (ozpeterballard at gmail.com)
Date: Tue, 20 Dec 2016 04:11:28 -0800 (PST)
Subject: Can't see numpy etc after upgrading Python on Ubuntu
In-Reply-To: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
References: <4e0e0511-cbcb-420a-8677-30c6ab6b8e3f@googlegroups.com>
Message-ID: <337280b1-59cb-48d5-995d-399853474e77@googlegroups.com>

OK thanks both of you. I didn't realise there was a distinction between system python and other (user/personal) python. Yes, python2.7.3 is still there in /usr/bin/python .

/usr/bin/python appears to be getting numpy and scipy from /usr/lib/python2.7/dist-packages . So I added that to PYTHONPATH and reran python (i.e. /usr/local/bin/python, which is 2.7.12). 

But now when I "import numpy" I get an error message which ends in:
 File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS4_AsUnicodeEscapeString

So for now I give up.

But the good news is that I have learned that my "old" python (2.7.3) is still on my machine. So for now I am renaming /usr/local/bin/python to /usr/local/bin/python2.7.12 , and going back to using python 2.7.3. 


From akbar at games2win.com  Tue Dec 20 07:28:19 2016
From: akbar at games2win.com (Akbar Shaikh)
Date: Tue, 20 Dec 2016 04:28:19 -0800 (PST)
Subject: Geting error using python 3.5 : a_token = r.json() mention below
Message-ID: <3bc1cb28-b689-4844-936f-91c08b4bf702@googlegroups.com>

import io
import csv
import glob
import os.path
import requests
import subprocess
import urllib.request
import json
import time
import xlwt
import xlrd
import datetime
from datetime import date, timedelta

def main():
        """ Run the whole toolchain for all accounts. """
        _clean()

        payload = {'client_id':'xxxx' ,
                    'client_secret':'xxxx',
                    'grant_type':'client_credentials'}
        headers1 = {
                'Content-Type':'application/json'
                }
        
        r = requests.post('https://auth.smaato.com/v2/auth/token/ HTTP/1.1',params=payload,auth=('username', 'password'),headers = headers1,)

        a_token = r.json()
        
        ACCESSTOKEN = a_token['access_token']
        print ('Bearer ' + ACCESSTOKEN)

        content2 = {
                    'client_id':'xxxx' ,
                    'client_secret':'xxxx',
                    'grant_type':'client_credentials',
                    'Authorization': 'Bearer' + ACCESSTOKEN,
                    'POST':'https://api.smaato.com/v1/reporting/ HTTP/1.1',
                    'Content-Type':'application/json',
                    'Host': 'api.smaato.com',
                    'criteria':{"dimension":"ApplicationId","child":"null"},
                    'kpi': {"clicks : true"},
                    'period':{"period_type":"fixed","start_date":"2016-12-7","end_date":"2016-12-7"}
                    }
        
        headers2 = {
                'client_id':'xxxx' ,
                'client_secret':'xxxx',
                'grant_type':'client_credentials',
                'Authorization': 'Bearer' + ACCESSTOKEN,
                'Username':'Username',
                'Password':'Password',
                'Content-Type': 'application/json',
                'Host': 'api.smaato.com',
                }
        s = requests.post('https://api.smaato.com/v1/reporting/',params=content2,auth=('username', 'password'), headers = headers2)
        print(s.content)
              
def _clean():
        """ Cleans old data files.
        """
        for f in glob.glob('./Numbers *.csv'):
                os.remove(f)
                
if '__main__' == __name__:
        main()
-------------------------------------------------------------------------
Error:  Traceback (most recent call last):
  File "C:\Users\xxxx\Desktop\Smaato Akbar.py", line 66, in <module>
    main()
  File "C:\Users\xxxx\Desktop\Smaato Akbar.py", line 28, in main
    a_token = r.json()
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\models.py", line 850, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


From ned at nedbatchelder.com  Tue Dec 20 10:37:48 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 20 Dec 2016 07:37:48 -0800 (PST)
Subject: Geting error using python 3.5 : a_token = r.json() mention below
In-Reply-To: <3bc1cb28-b689-4844-936f-91c08b4bf702@googlegroups.com>
References: <3bc1cb28-b689-4844-936f-91c08b4bf702@googlegroups.com>
Message-ID: <84dedc86-6c9e-4568-84ca-57e0e678c178@googlegroups.com>

On Tuesday, December 20, 2016 at 7:28:47 AM UTC-5, Akbar Shaikh wrote:
> import io
> import csv
> import glob
> import os.path
> import requests
> import subprocess
> import urllib.request
> import json
> import time
> import xlwt
> import xlrd
> import datetime
> from datetime import date, timedelta
> 
> def main():
>         """ Run the whole toolchain for all accounts. """
>         _clean()
> 
>         payload = {'client_id':'xxxx' ,
>                     'client_secret':'xxxx',
>                     'grant_type':'client_credentials'}
>         headers1 = {
>                 'Content-Type':'application/json'
>                 }
>         
>         r = requests.post('https://auth.smaato.com/v2/auth/token/ HTTP/1.1',params=payload,auth=('username', 'password'),headers = headers1,)
> 
>         a_token = r.json()
>         
>         ACCESSTOKEN = a_token['access_token']
>         print ('Bearer ' + ACCESSTOKEN)
> 
>         content2 = {
>                     'client_id':'xxxx' ,
>                     'client_secret':'xxxx',
>                     'grant_type':'client_credentials',
>                     'Authorization': 'Bearer' + ACCESSTOKEN,
>                     'POST':'https://api.smaato.com/v1/reporting/ HTTP/1.1',
>                     'Content-Type':'application/json',
>                     'Host': 'api.smaato.com',
>                     'criteria':{"dimension":"ApplicationId","child":"null"},
>                     'kpi': {"clicks : true"},
>                     'period':{"period_type":"fixed","start_date":"2016-12-7","end_date":"2016-12-7"}
>                     }
>         
>         headers2 = {
>                 'client_id':'xxxx' ,
>                 'client_secret':'xxxx',
>                 'grant_type':'client_credentials',
>                 'Authorization': 'Bearer' + ACCESSTOKEN,
>                 'Username':'Username',
>                 'Password':'Password',
>                 'Content-Type': 'application/json',
>                 'Host': 'api.smaato.com',
>                 }
>         s = requests.post('https://api.smaato.com/v1/reporting/',params=content2,auth=('username', 'password'), headers = headers2)
>         print(s.content)
>               
> def _clean():
>         """ Cleans old data files.
>         """
>         for f in glob.glob('./Numbers *.csv'):
>                 os.remove(f)
>                 
> if '__main__' == __name__:
>         main()
> -------------------------------------------------------------------------
> Error:  Traceback (most recent call last):
>   File "C:\Users\xxxx\Desktop\Smaato Akbar.py", line 66, in <module>
>     main()
>   File "C:\Users\xxxx\Desktop\Smaato Akbar.py", line 28, in main
>     a_token = r.json()
>   File "C:\Users\xxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\models.py", line 850, in json
>     return complexjson.loads(self.text, **kwargs)
>   File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\__init__.py", line 516, in loads
>     return _default_decoder.decode(s)
>   File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\decoder.py", line 374, in decode
>     obj, end = self.raw_decode(s)
>   File "C:\Users\xxxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\simplejson\decoder.py", line 404, in raw_decode
>     return self.scan_once(s, idx=_w(s, idx).end())
> simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


That error means you are trying to decode an empty string. Most likely,
your request failed, and there's a status code that will give you a clue
as to why it failed.  You should check the success of the request before
trying to use the results.

>From the docs (http://docs.python-requests.org/en/master/user/quickstart/#json-response-content):

> To check that a request is successful, use r.raise_for_status() or check
> r.status_code is what you expect.

--Ned.


From uri at speedy.net  Tue Dec 20 11:57:57 2016
From: uri at speedy.net (Uri Even-Chen)
Date: Tue, 20 Dec 2016 18:57:57 +0200
Subject: ImportError: The ``fake-factory`` package is now called ``Faker``.
Message-ID: <CAMQ2MsEjOtyUryABSr2qnN17C-B=jZM2szoR1W-fVGQxmBB4fQ@mail.gmail.com>

Hi, we get this error with Python 3.4 and 3.5:
https://travis-ci.org/urievenchen/speedy-net/jobs/185497863


----------------------------------------------------------------------

ImportError: Failed to import test module:
speedy.net.accounts.tests.test_factories

Traceback (most recent call last):

  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 428,
in _find_test_path

    module = self._get_module_from_name(name)

  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 369,
in _get_module_from_name

    __import__(name)

  File "/home/travis/build/urievenchen/speedy-net/speedy/net/accounts/tests/test_factories.py",
line 4, in <module>

    import factory

  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/factory/__init__.py",
line 46, in <module>

    from .faker import Faker

  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/factory/faker.py",
line 41, in <module>

    import faker

  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/faker/__init__.py",
line 7, in <module>

    raise ImportError(error)

ImportError: The ``fake-factory`` package is now called ``Faker``.

Please update your requirements.


What is the problem? We didn't update our requirements recently.

Thanks,
Uri.

*Uri Even-Chen*
[image: photo] Phone: +972-54-3995700
Email: uri at speedy.net
Website: http://www.speedysoftware.com/uri/en/
<http://www.facebook.com/urievenchen>  <http://plus.google.com/+urievenchen>
  <http://www.linkedin.com/in/urievenchen>  <http://github.com/urievenchen>
<http://twitter.com/urievenchen>


From mail at williammayor.co.uk  Tue Dec 20 12:11:28 2016
From: mail at williammayor.co.uk (William Mayor)
Date: Tue, 20 Dec 2016 17:11:28 +0000
Subject: ImportError: The ``fake-factory`` package is now called ``Faker``.
In-Reply-To: <CAMQ2MsEjOtyUryABSr2qnN17C-B=jZM2szoR1W-fVGQxmBB4fQ@mail.gmail.com>
References: <CAMQ2MsEjOtyUryABSr2qnN17C-B=jZM2szoR1W-fVGQxmBB4fQ@mail.gmail.com>
Message-ID: <E315C06B-B4E6-4A2A-98C2-0708B839813E@williammayor.co.uk>

I came across this error this morning.

In my case I was running a script that did this: ?pip install fake-factory?

The install worked properly but then I couldn?t import the library (with the same error message as you).

I had to update the pip install command to say ?pip install Faker? and then everything worked as before.

The authors appear to have forced a name change: https://github.com/joke2k/faker/blob/fake-factory/faker/__init__.py <https://github.com/joke2k/faker/blob/fake-factory/faker/__init__.py>

I?ve just checked and if you pin your requirements (e.g. pip install fake-factory==0.7.4) then it still works.


> On 20 Dec 2016, at 16:57, Uri Even-Chen <uri at speedy.net> wrote:
> 
> Hi, we get this error with Python 3.4 and 3.5:
> https://travis-ci.org/urievenchen/speedy-net/jobs/185497863
> 
> 
> ----------------------------------------------------------------------
> 
> ImportError: Failed to import test module:
> speedy.net.accounts.tests.test_factories
> 
> Traceback (most recent call last):
> 
>  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 428,
> in _find_test_path
> 
>    module = self._get_module_from_name(name)
> 
>  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 369,
> in _get_module_from_name
> 
>    __import__(name)
> 
>  File "/home/travis/build/urievenchen/speedy-net/speedy/net/accounts/tests/test_factories.py",
> line 4, in <module>
> 
>    import factory
> 
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/factory/__init__.py",
> line 46, in <module>
> 
>    from .faker import Faker
> 
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/factory/faker.py",
> line 41, in <module>
> 
>    import faker
> 
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/faker/__init__.py",
> line 7, in <module>
> 
>    raise ImportError(error)
> 
> ImportError: The ``fake-factory`` package is now called ``Faker``.
> 
> Please update your requirements.
> 
> 
> What is the problem? We didn't update our requirements recently.
> 
> Thanks,
> Uri.
> 
> *Uri Even-Chen*
> [image: photo] Phone: +972-54-3995700
> Email: uri at speedy.net
> Website: http://www.speedysoftware.com/uri/en/
> <http://www.facebook.com/urievenchen>  <http://plus.google.com/+urievenchen>
>  <http://www.linkedin.com/in/urievenchen>  <http://github.com/urievenchen>
> <http://twitter.com/urievenchen>
> -- 
> https://mail.python.org/mailman/listinfo/python-list



From flebber.crue at gmail.com  Tue Dec 20 13:58:02 2016
From: flebber.crue at gmail.com (Sayth Renshaw)
Date: Tue, 20 Dec 2016 10:58:02 -0800 (PST)
Subject: windows utf8 & lxml
In-Reply-To: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
References: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
Message-ID: <cbabc9e8-83f9-4705-b0b4-d96142f2a67b@googlegroups.com>

Possibly i will have to use a different method from lxml like this. 
http://stackoverflow.com/a/29057244/461887

Sayth


From mr at e-wrobel.pl  Tue Dec 20 16:04:40 2016
From: mr at e-wrobel.pl (Mr. Wrobel)
Date: Tue, 20 Dec 2016 22:04:40 +0100
Subject: Metaclasses - magic functions
Message-ID: <o3c6d8$sru$1@news.icm.edu.pl>

Hi,

Quick question, can anybody tell me when to use __init__ instead of 
__new__ in meta programming?

I see that __new__ can be used mostly when I want to manipulate with 
class variables that are stored into dictionary.

But when to use __init__? Any example?

Thanx,
M


From saxri89 at gmail.com  Tue Dec 20 16:46:44 2016
From: saxri89 at gmail.com (Xristos Xristoou)
Date: Tue, 20 Dec 2016 13:46:44 -0800 (PST)
Subject: topology rules in python
Message-ID: <adf9c9fa-1e08-4c4b-a5b9-69752b061386@googlegroups.com>

I have a PostGIS database with shapefiles lines, polygons and points and I want to create a topology rules with python.
Any idea how to do that ?some packages ?

I like some easy way to do that because I am newbie but its OK.

Some topology rules when I want.
shapefile must not have gaps
shapefile must not have overlaps
shapefile must not overlap with shapefile2
invalid geometries


From ian.g.kelly at gmail.com  Tue Dec 20 18:26:04 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 20 Dec 2016 16:26:04 -0700
Subject: Metaclasses - magic functions
In-Reply-To: <o3c6d8$sru$1@news.icm.edu.pl>
References: <o3c6d8$sru$1@news.icm.edu.pl>
Message-ID: <CALwzidmdR-8CyAcDQN7NrncpQLtk8MsbhTPW0jFpyfm5xiOySw@mail.gmail.com>

On Tue, Dec 20, 2016 at 2:04 PM, Mr. Wrobel <mr at e-wrobel.pl> wrote:
> Hi,
>
> Quick question, can anybody tell me when to use __init__ instead of __new__
> in meta programming?
>
> I see that __new__ can be used mostly when I want to manipulate with class
> variables that are stored into dictionary.
>
> But when to use __init__? Any example?

__new__ is useful because of its power. You can use it to do things
like alter the name of the class or its base classes, or modify class
attributes before the class is created. The latter is interesting
mostly because it allows you to set the __slots__ or do something
interesting with the __prepare__ hook (although the only interesting
thing I've ever seen done with __prepare__ is to use an OrderedDict to
preserve the the definition order of the class attributes; now that
Python 3.6 does this by default it's probably obsolete).

__init__ is simpler than __new__. If you don't need to do any of the
fancy stuff above, you should probably use __init__ instead.

I can't think of any reason why one would ever want to use both on a metaclass.


From ben+python at benfinney.id.au  Tue Dec 20 18:39:33 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 21 Dec 2016 10:39:33 +1100
Subject: Metaclasses - magic functions
References: <o3c6d8$sru$1@news.icm.edu.pl>
Message-ID: <85a8bqxj9m.fsf@benfinney.id.au>

"Mr. Wrobel" <mr at e-wrobel.pl> writes:

> Quick question, can anybody tell me when to use __init__ instead of
> __new__ in meta programming?

Use ?__new__? to do the work of *creating* one instance from nothing;
allocating the storage, determining the type, etc. ? anything that will
be *the same* for every instance. ?__new__? is the constructor.

Use ?__init__? to do the work of *configuring* one instance, after it
already exists. Any attributes that are special to an instance should be
manipulated in the ?__init__? method. ?__init__? is the initialiser.

Because Python has a powerful type system built in, you generally have
little or nothing to do in ?__new__?, and more work in ?__init__? for
each instance.


That's a general answer because you haven't said what your goal is. What
is it you want to do with types?

-- 
 \      ?Fur coats made for the ladies from their own skin.? ?furrier, |
  `\                                                            Sweden |
_o__)                                                                  |
Ben Finney



From ethan at stoneleaf.us  Tue Dec 20 18:56:10 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 20 Dec 2016 15:56:10 -0800
Subject: Metaclasses - magic functions
In-Reply-To: <CALwzidmdR-8CyAcDQN7NrncpQLtk8MsbhTPW0jFpyfm5xiOySw@mail.gmail.com>
References: <o3c6d8$sru$1@news.icm.edu.pl>
 <CALwzidmdR-8CyAcDQN7NrncpQLtk8MsbhTPW0jFpyfm5xiOySw@mail.gmail.com>
Message-ID: <5859C51A.70603@stoneleaf.us>

On 12/20/2016 03:26 PM, Ian Kelly wrote:

> ... The latter is interesting mostly because it allows you to set the
> __slots__ or do something interesting with the __prepare__ hook
> (although the only interesting thing I've ever seen done with
> __prepare__ is to use an OrderedDict to preserve the the definition
> order of the class attributes; now that Python 3.6 does this by default
>  it's probably obsolete).

__prepare__ is not obsolete.  Enum uses it to set a custom dictionary which tracks the new members, etc.

--
~Ethan~


From ethan at stoneleaf.us  Tue Dec 20 20:51:15 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 20 Dec 2016 17:51:15 -0800
Subject: Metaclasses - magic functions
In-Reply-To: <85a8bqxj9m.fsf@benfinney.id.au>
References: <o3c6d8$sru$1@news.icm.edu.pl> <85a8bqxj9m.fsf@benfinney.id.au>
Message-ID: <5859E013.8010108@stoneleaf.us>

On 12/20/2016 03:39 PM, Ben Finney wrote:
> "Mr. Wrobel" writes:

>> Quick question, can anybody tell me when to use __init__ instead of
>> __new__ in meta programming?
>
> Use ?__new__? to do the work of *creating* one instance from nothing;
> allocating the storage, determining the type, etc. ? anything that will
> be *the same* for every instance. ?__new__? is the constructor.
>
> Use ?__init__? to do the work of *configuring* one instance, after it
> already exists. Any attributes that are special to an instance should be
> manipulated in the ?__init__? method. ?__init__? is the initialiser.

That sounds like general object creation/class advice, which as a general
guideline is okay, but don't feel like it's the most important thing.

I only use `__new__` when the object being created is (or is based on)
an immutable type; otherwise I use `__init__`.  Likewise, if I'm using
`__new__` then I do all my configuration in `__new__` unless I have a
really good reason not to (such as making it easier for subclasses to
modify/skip `__init__`).

As far as metaclasses go... the only time I recall writing an `__init__`
for a metaclass was to strip off the extra arguments so `type.__init__`
wouldn't fail.

--
~Ethan~


From malik.a.rumi at gmail.com  Tue Dec 20 21:37:35 2016
From: malik.a.rumi at gmail.com (Malik Rumi)
Date: Tue, 20 Dec 2016 18:37:35 -0800 (PST)
Subject: Pyvenv puts both Python 2 and Python 3 in the same environment.
 Shocked!
Message-ID: <fae1943f-4713-49e1-bd4f-ed29a8c57027@googlegroups.com>

I just created a new venv using pyvenv from a 2.7 install. Now I am shocked to see that I can get both 2.7 and 3.4 in this same venv:

(memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
(memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python3
Python 3.4.2 (default, Apr 17 2015, 18:47:05) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

I did not even know this was possible. Doesn?t that defeat the purpose? More to the point, how do I make sure I am using the right one? If I want the interpreter, that?s easy. But what happens when I want to install and use a program like Django? How do I control which interpreter Django uses? I?ve been through the official docs a couple of times today, but detailed explanations of pyvenv, let alone this dual version feature, have not been found. If you can point me to a good one, please do. Meanwhile...

I hoped maybe using different versions of pip would solve the problem, but pip2 list and pip3 list are identical ? hence, the very problem I thought venv?s were supposed to avoid. 

Thanks


From steve+comp.lang.python at pearwood.info  Tue Dec 20 23:06:42 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 21 Dec 2016 15:06:42 +1100
Subject: Pyvenv puts both Python 2 and Python 3 in the same environment.
 Shocked!
References: <fae1943f-4713-49e1-bd4f-ed29a8c57027@googlegroups.com>
Message-ID: <5859ffd4$0$1600$c3e8da3$5496439d@news.astraweb.com>

On Wednesday 21 December 2016 13:37, Malik Rumi wrote:

> I just created a new venv using pyvenv from a 2.7 install. Now I am shocked
> to see that I can get both 2.7 and 3.4 in this same venv:

I'm not an expert on pyvenv, but my guess is this:

Would you expect a command like "ls" or "grep" to continue to work from inside 
a venv? I expect you would say Yes.

How about "perl"?

If your venv has shadowed python, using the venv instead of the system python, 
there's no reason to expect that python3 will be any different than the same 
command outside of the venv. It will just run the same executable found in the 
PATH as normal, just like (say) "perl" or "grep".

> I did not even know this was possible. Doesn?t that defeat the purpose? More
> to the point, how do I make sure I am using the right one? 

Run this command from the shell prompt:

    which python

and that should tell you which executable you are getting.

(I think that should work inside a venv.)


> If I want the
> interpreter, that?s easy. But what happens when I want to install and use a
> program like Django? How do I control which interpreter Django uses? I?ve
> been through the official docs a couple of times today, but detailed
> explanations of pyvenv, let alone this dual version feature, have not been
> found. If you can point me to a good one, please do. Meanwhile...

How do you control which interpreter Django uses outside of a venv?




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From cs at zip.com.au  Tue Dec 20 23:14:32 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 21 Dec 2016 15:14:32 +1100
Subject: Pyvenv puts both Python 2 and Python 3 in the same environment.
 Shocked!
In-Reply-To: <fae1943f-4713-49e1-bd4f-ed29a8c57027@googlegroups.com>
References: <fae1943f-4713-49e1-bd4f-ed29a8c57027@googlegroups.com>
Message-ID: <20161221041432.GA61496@cskk.homeip.net>

On 20Dec2016 18:37, Malik Rumi <malik.a.rumi at gmail.com> wrote:
>I just created a new venv using pyvenv from a 2.7 install. Now I am shocked to see that I can get both 2.7 and 3.4 in this same venv:
>
>(memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python
>Python 2.7.12 (default, Nov 19 2016, 06:48:10)
>[GCC 5.4.0 20160609] on linux2
>Type "help", "copyright", "credits" or "license" for more information.
>>>> exit()
>(memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python3
>Python 3.4.2 (default, Apr 17 2015, 18:47:05)
>[GCC 4.8.2] on linux
>Type "help", "copyright", "credits" or "license" for more information.
>
>I did not even know this was possible. Doesn?t that defeat the purpose? More 
>to the point, how do I make sure I am using the right one? If I want the 
>interpreter, that?s easy. But what happens when I want to install and use a 
>program like Django? How do I control which interpreter Django uses? I?ve been 
>through the official docs a couple of times today, but detailed explanations 
>of pyvenv, let alone this dual version feature, have not been found. If you 
>can point me to a good one, please do. Meanwhile...

I suspect you're confused. What do:

  which python

and

  which python3

say? Creating a virtual env does not do anything in particular to your 
environment. It provides a distinct environment to work in, but you do need to 
take a specific action to use it.

I'd inspect sys.path in each of your pythons:

  >>> import sys
  >>> sys.path

Watch on my own Mac:

  % /opt/local/bin/python3
  Python 3.5.2 (default, Oct 11 2016, 15:01:29)
  [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sys
  >>> sys.path
  ['', '/Users/cameron/lib/python', '/Users/cameron/rc/python', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python35.zip', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages']

versus this:

  % ~/var/venv/3/bin/python3
  Python 3.5.2 (default, Oct 11 2016, 15:01:29)
  [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sys
  >>> sys.path
  ['', '/Users/cameron/lib/python', '/Users/cameron/rc/python', 
  '/Users/cameron/var/venv/3/lib/python35.zip', 
  '/Users/cameron/var/venv/3/lib/python3.5', 
  '/Users/cameron/var/venv/3/lib/python3.5/plat-darwin', 
  '/Users/cameron/var/venv/3/lib/python3.5/lib-dynload', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin', 
  '/Users/cameron/var/venv/3/lib/python3.5/site-packages', 
  '/Users/cameron/var/venv/3/lib/python3.5/site-packages/llfuse-1.1.1-py3.5-macosx-10.11-x86_64.egg', 
  '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages']

so you can see that the "native" python3 has one sys.path, and the one from my 
local python 3 virtualenv has a different one.

Check which pythons you're actually invoking, and what their respective 
sys.path values are.

Cheers,
Cameron Simpson <cs at zip.com.au>


From Bernd.Nawothnig at t-online.de  Wed Dec 21 01:11:34 2016
From: Bernd.Nawothnig at t-online.de (Bernd Nawothnig)
Date: Wed, 21 Dec 2016 07:11:34 +0100
Subject: topology rules in python
References: <adf9c9fa-1e08-4c4b-a5b9-69752b061386@googlegroups.com>
Message-ID: <m1jpid-v43.ln1@bernd.nawothnig.dialin.t-online.de>

On 2016-12-20, Xristos Xristoou wrote:
> I have a PostGIS database with shapefiles lines, polygons and points
> and I want to create a topology rules with python. Any idea how to do
> that ?some packages ?

http://www.gdal.org/

or:

pip install gdal




Bernd

-- 
no time toulouse


From paul.nospam at rudin.co.uk  Wed Dec 21 02:21:54 2016
From: paul.nospam at rudin.co.uk (Paul Rudin)
Date: Wed, 21 Dec 2016 07:21:54 +0000
Subject: Pyvenv puts both Python 2 and Python 3 in the same environment.
 Shocked!
References: <fae1943f-4713-49e1-bd4f-ed29a8c57027@googlegroups.com>
Message-ID: <871sx1n3vx.fsf@rudin.co.uk>

Malik Rumi <malik.a.rumi at gmail.com> writes:

> I just created a new venv using pyvenv from a 2.7 install. Now I am shocked to
> see that I can get both 2.7 and 3.4 in this same venv:
>
> (memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python
> Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
> [GCC 5.4.0 20160609] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> exit()
> (memory) malikarumi at Tetuoan2:~/Projects/cannon/New2.7Projects/memory$ python3
> Python 3.4.2 (default, Apr 17 2015, 18:47:05) 
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.

A venv doesn't remove everything from your path. Presumably python3 is
some system installed python. What does "which python3" say?

If you just invoke "python" then you'll get the venv one.




From flebber.crue at gmail.com  Wed Dec 21 04:03:48 2016
From: flebber.crue at gmail.com (Sayth Renshaw)
Date: Wed, 21 Dec 2016 01:03:48 -0800 (PST)
Subject: windows utf8 & lxml
In-Reply-To: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
References: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
Message-ID: <36827d43-9c4e-4916-9dd1-9ffb1acd06c9@googlegroups.com>

On Tuesday, 20 December 2016 22:54:03 UTC+11, Sayth Renshaw  wrote:
> Hi 
> 
> I have been trying to get a script to work on windows that works on mint. The key blocker has been utf8 errors, most of which I have solved.
> 
> Now however the last error I am trying to overcome, the solution appears to be to use the .decode('windows-1252') to correct an ascii error.
> 
> I am using lxml to read my content and decode is not supported are there any known ways to read with lxml and fix unicode faults?
> 
> The key part of my script is 
> 
>         for content in roots:
>             utf8_parser = etree.XMLParser(encoding='utf-8')
>             fix_ascii = utf8_parser.decode('windows-1252')
>             mytree = etree.fromstring(
>                 content.read().encode('utf-8'), parser=fix_ascii)
> 
> Without the added .decode my code looks like
> 
>         for content in roots:
>             utf8_parser = etree.XMLParser(encoding='utf-8')
>             mytree = etree.fromstring(
>                 content.read().encode('utf-8'), parser=utf8_parser)
> 
> However doing it in such a fashion returns this error:
> 
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
> Which I found this SO for http://stackoverflow.com/a/29217546/461887 but cannot seem to implement with lxml.
> 
> Ideas?
> 
> Sayth

Why is windows so hard. Sort of running out of ideas, tried methods in the docs SO etc.

Currently

        for xml_data in roots:
            parser_xml = etree.XMLParser()
            mytree = etree.parse(xml_data, parser_xml)

Returns
C:\Users\Sayth\Anaconda3\envs\race\python.exe C:/Users/Sayth/PycharmProjects/bs4race/race.py data/ -e *.xml
Traceback (most recent call last):
  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 100, in <module>
    data_attr(rootObs)
  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 55, in data_attr
    mytree = etree.parse(xml_data, parser_xml)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src\lxml\lxml.etree.c:81110)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src\lxml\lxml.etree.c:118109)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src\lxml\lxml.etree.c:118392)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src\lxml\lxml.etree.c:117180)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src\lxml\lxml.etree.c:111907)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:105102)
  File "src/lxml/parser.pxi", line 702, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:106769)
  File "src/lxml/lxml.etree.pyx", line 324, in lxml.etree._ExceptionContext._raise_if_stored (src\lxml\lxml.etree.c:12074)
  File "src/lxml/parser.pxi", line 373, in lxml.etree._FileReaderContext.copyToBuffer (src\lxml\lxml.etree.c:102431)
io.UnsupportedOperation: read

Process finished with exit code 1

Thoughts?

Sayth


From __peter__ at web.de  Wed Dec 21 04:36:39 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 21 Dec 2016 10:36:39 +0100
Subject: windows utf8 & lxml
References: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
 <36827d43-9c4e-4916-9dd1-9ffb1acd06c9@googlegroups.com>
Message-ID: <o3dif4$5g7$1@blaine.gmane.org>

Sayth Renshaw wrote:

> On Tuesday, 20 December 2016 22:54:03 UTC+11, Sayth Renshaw  wrote:
>> Hi
>> 
>> I have been trying to get a script to work on windows that works on mint.
>> The key blocker has been utf8 errors, most of which I have solved.
>> 
>> Now however the last error I am trying to overcome, the solution appears
>> to be to use the .decode('windows-1252') to correct an ascii error.
>> 
>> I am using lxml to read my content and decode is not supported are there
>> any known ways to read with lxml and fix unicode faults?
>> 
>> The key part of my script is
>> 
>>         for content in roots:
>>             utf8_parser = etree.XMLParser(encoding='utf-8')
>>             fix_ascii = utf8_parser.decode('windows-1252')
>>             mytree = etree.fromstring(
>>                 content.read().encode('utf-8'), parser=fix_ascii)
>> 
>> Without the added .decode my code looks like
>> 
>>         for content in roots:
>>             utf8_parser = etree.XMLParser(encoding='utf-8')
>>             mytree = etree.fromstring(
>>                 content.read().encode('utf-8'), parser=utf8_parser)
>> 
>> However doing it in such a fashion returns this error:
>> 
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0:
>> invalid start byte Which I found this SO for
>> http://stackoverflow.com/a/29217546/461887 but cannot seem to implement
>> with lxml.
>> 
>> Ideas?
>> 
>> Sayth
> 
> Why is windows so hard. 

I don't think this has anything to do with the OS. Your lxml_data is 
probably not what you think it is. Compare:

$ python3
Python 3.4.3 (default, Nov 17 2016, 01:08:31) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import lxml.etree
>>> lxml.etree.parse(sys.stdout)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml.etree.pyx", line 3239, in lxml.etree.parse 
(src/lxml/lxml.etree.c:69955)
  File "parser.pxi", line 1769, in lxml.etree._parseDocument 
(src/lxml/lxml.etree.c:102257)
  File "parser.pxi", line 1789, in lxml.etree._parseFilelikeDocument 
(src/lxml/lxml.etree.c:102516)
  File "parser.pxi", line 1684, in lxml.etree._parseDocFromFilelike 
(src/lxml/lxml.etree.c:101442)
  File "parser.pxi", line 1134, in 
lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:97069)
  File "parser.pxi", line 582, in 
lxml.etree._ParserContext._handleParseResultDoc 
(src/lxml/lxml.etree.c:91275)
  File "parser.pxi", line 679, in lxml.etree._handleParseResult 
(src/lxml/lxml.etree.c:92426)
  File "lxml.etree.pyx", line 327, in 
lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:10196)
  File "parser.pxi", line 373, in lxml.etree._FileReaderContext.copyToBuffer 
(src/lxml/lxml.etree.c:89083)
io.UnsupportedOperation: not readable

That looks similar to what you get.

> Sort of running out of ideas, tried methods in the
> docs SO etc.
> 
> Currently
> 
>         for xml_data in roots:
>             parser_xml = etree.XMLParser()
>             mytree = etree.parse(xml_data, parser_xml)
> 
> Returns
> C:\Users\Sayth\Anaconda3\envs\race\python.exe
> C:/Users/Sayth/PycharmProjects/bs4race/race.py data/ -e *.xml Traceback
> (most recent call last):
>   File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 100, in
>   <module>
>     data_attr(rootObs)
>   File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 55, in
>   data_attr
>     mytree = etree.parse(xml_data, parser_xml)
>   File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse
>   (src\lxml\lxml.etree.c:81110) File "src/lxml/parser.pxi", line 1832, in
>   lxml.etree._parseDocument (src\lxml\lxml.etree.c:118109) File
>   "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument
>   (src\lxml\lxml.etree.c:118392) File "src/lxml/parser.pxi", line 1747, in
>   lxml.etree._parseDocFromFilelike (src\lxml\lxml.etree.c:117180) File
>   "src/lxml/parser.pxi", line 1162, in
>   lxml.etree._BaseParser._parseDocFromFilelike
>   (src\lxml\lxml.etree.c:111907) File "src/lxml/parser.pxi", line 595, in
>   lxml.etree._ParserContext._handleParseResultDoc
>   (src\lxml\lxml.etree.c:105102) File "src/lxml/parser.pxi", line 702, in
>   lxml.etree._handleParseResult (src\lxml\lxml.etree.c:106769) File
>   "src/lxml/lxml.etree.pyx", line 324, in
>   lxml.etree._ExceptionContext._raise_if_stored
>   (src\lxml\lxml.etree.c:12074) File "src/lxml/parser.pxi", line 373, in
>   lxml.etree._FileReaderContext.copyToBuffer
>   (src\lxml\lxml.etree.c:102431)
> io.UnsupportedOperation: read
> 
> Process finished with exit code 1
> 
> Thoughts?
> 
> Sayth




From flebber.crue at gmail.com  Wed Dec 21 04:47:12 2016
From: flebber.crue at gmail.com (Sayth Renshaw)
Date: Wed, 21 Dec 2016 01:47:12 -0800 (PST)
Subject: for loop iter next if file bad
Message-ID: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>

Hi

I am looping a list of files and want to skip any empty files.
I get an error that str is not an iterator which I sought of understand but can't see a workaround for.

How do I make this an iterator so I can use next on the file if my test returns true.

Currently my code is.

for dir_path, subdir_list, file_list in os.walk(my_dir):
    for name_pattern in file_list:
        full_path = os.path.join(dir_path, name_pattern)


def return_files(file_list):
    """
    Take a list of files and return file when called.

    Calling function to supply attributes
    """
    for file in file_list:
        with open(os.path.join(dir_path, file), 'rb') as fd:
            if os.stat(fd.name).st_size == 0:
                next(file)
            else:
                yield fd

Exact error is:

C:\Users\Sayth\Anaconda3\envs\race\python.exe C:/Users/Sayth/PycharmProjects/bs4race/race.py data/ -e *.xml
Traceback (most recent call last):
  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 98, in <module>
    data_attr(rootObs)
  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 51, in data_attr
    for xml_data in roots:
  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 32, in return_files
    next(file)
TypeError: 'str' object is not an iterator

Sayth


From Joaquin.Alzola at lebara.com  Wed Dec 21 04:55:45 2016
From: Joaquin.Alzola at lebara.com (Joaquin Alzola)
Date: Wed, 21 Dec 2016 09:55:45 +0000
Subject: for loop iter next if file bad
In-Reply-To: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>
References: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>
Message-ID: <DB5PR07MB08060C0A57E58A6ADB230567F0930@DB5PR07MB0806.eurprd07.prod.outlook.com>


>def return_files(file_list):
>    """
>    Take a list of files and return file when called.
>
>    Calling function to supply attributes
>    """
>    for file in file_list:
>        with open(os.path.join(dir_path, file), 'rb') as fd:
>            if os.stat(fd.name).st_size == 0:
>                next(file)
>            else:
>                yield fd

>Exact error is:

>C:\Users\Sayth\Anaconda3\envs\race\python.exe C:/Users/Sayth/PycharmProjects/bs4race/race.py data/ -e *.xml
>Traceback (most recent call last):
 > File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 98, in <module>
>    data_attr(rootObs)
>  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 51, in data_attr
>    for xml_data in roots:
>  File "C:/Users/Sayth/PycharmProjects/bs4race/race.py", line 32, in return_files
>    next(file)
>TypeError: 'str' object is not an iterator

The iterator is file_list not file.
File is a str. As the exception mentions.

I suppose you want there a "continue" to iterate to the next value in the file_list



This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.


From rosuav at gmail.com  Wed Dec 21 04:59:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 21 Dec 2016 20:59:58 +1100
Subject: for loop iter next if file bad
In-Reply-To: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>
References: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>
Message-ID: <CAPTjJmpM+TUHzWG3NEhnTMLeE_O5nzwNp9h0AYv9fGxgFK43Sg@mail.gmail.com>

On Wed, Dec 21, 2016 at 8:47 PM, Sayth Renshaw <flebber.crue at gmail.com> wrote:
> def return_files(file_list):
>     """
>     Take a list of files and return file when called.
>
>     Calling function to supply attributes
>     """
>     for file in file_list:
>         with open(os.path.join(dir_path, file), 'rb') as fd:
>             if os.stat(fd.name).st_size == 0:
>                 next(file)
>             else:
>                 yield fd

"next" doesn't do what you think it does - it tries to step the thing
you give it, as an iterator. I think you might want "continue"?

ChrisA


From flebber.crue at gmail.com  Wed Dec 21 05:06:52 2016
From: flebber.crue at gmail.com (Sayth Renshaw)
Date: Wed, 21 Dec 2016 02:06:52 -0800 (PST)
Subject: for loop iter next if file bad
In-Reply-To: <mailman.191.1482314407.2322.python-list@python.org>
References: <a1f5003d-3e8c-4359-a32e-f51f6cd89d51@googlegroups.com>
 <CAPTjJmpM+TUHzWG3NEhnTMLeE_O5nzwNp9h0AYv9fGxgFK43Sg@mail.gmail.com>
 <mailman.191.1482314407.2322.python-list@python.org>
Message-ID: <1485b43a-c170-4e83-91e7-7342639b65ed@googlegroups.com>

Ah yes. Thanks ChrisA

http://www.tutorialspoint.com/python/python_loop_control.htm

The continue Statement:
The continue statement in Python returns the control to the beginning of the while loop. The continue statement rejects all the remaining statements in the current iteration of the loop and moves the control back to the top of the loop.

The continue statement can be used in both while and for loops.

Sayth


From ethan at stoneleaf.us  Wed Dec 21 12:35:23 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 21 Dec 2016 09:35:23 -0800
Subject: SQLAlchemy and Postgres
Message-ID: <585ABD5B.7000301@stoneleaf.us>

There's a question over on SO [1] asking about an interaction between
SQLAlchemy and postgres which may be related to an SQLA upgrade from
1.0 to 1.1 and the generation of a check clause

Sadly, I don't have any experience with SQLAlchemy -- anybody here
want to take a crack at it?

--
~Ethan~

[1] http://stackoverflow.com/q/41258376/208880


From skybuck2000 at hotmail.com  Wed Dec 21 13:10:23 2016
From: skybuck2000 at hotmail.com (skybuck2000 at hotmail.com)
Date: Wed, 21 Dec 2016 10:10:23 -0800 (PST)
Subject: [OT] "Invisible posts", was: Best attack order for groups of
 numbers trying to destroy each other, given a victory chance for number to
 number attack.
In-Reply-To: <mailman.88.1481804207.2322.python-list@python.org>
References: <51a8f559-84f1-4eb1-85ad-3700368d5e65@googlegroups.com>
 <50ff1854-ba2e-41d0-bd92-4f1809418fc3@googlegroups.com>
 <o2u1iu$und$1@blaine.gmane.org>
 <mailman.88.1481804207.2322.python-list@python.org>
Message-ID: <1dd5ea5c-dcc7-4c36-bf80-341c278af5a3@googlegroups.com>

On Thursday, December 15, 2016 at 1:17:10 PM UTC+1, Peter Otten wrote:
> skybuck2000 at hotmail.com wrote:
> 
> > I received a reply from somebody on my ISP newsserver. Apperently his
> > reply is not visible on google groups. I wonder why, maybe it's a banned
> > troll or something, but perhaps not.
> 
> No, that's Dennis Lee Bieber who doesn't want his posts to be kept, and 
> seems to be the only one to post here with the X-No-Archive flag set.

I find this weird behaviour by google newsgroup.

At least google newsgroups could show his postings for something more reasonable like 30 days or something, instead of not showing it at all...

Totally weird. What's the point in posting then if software would not display it at all ? ;)


From fabien.maussion at gmail.com  Wed Dec 21 14:53:36 2016
From: fabien.maussion at gmail.com (Fabien)
Date: Wed, 21 Dec 2016 20:53:36 +0100
Subject: topology rules in python
References: <adf9c9fa-1e08-4c4b-a5b9-69752b061386@googlegroups.com>
 <m1jpid-v43.ln1@bernd.nawothnig.dialin.t-online.de>
Message-ID: <o3emju$ttm$1@gioia.aioe.org>

On 12/21/2016 07:11 AM, Bernd Nawothnig wrote:
> On 2016-12-20, Xristos Xristoou wrote:
>> I have a PostGIS database with shapefiles lines, polygons and points
>> and I want to create a topology rules with python. Any idea how to do
>> that ?some packages ?
> http://www.gdal.org/
>
> or:
>
> pip install gdal

also: shapely, geopandas




From python at deborahswanson.net  Wed Dec 21 19:55:18 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Wed, 21 Dec 2016 16:55:18 -0800
Subject: Mergesort problem
In-Reply-To: <36827d43-9c4e-4916-9dd1-9ffb1acd06c9@googlegroups.com>
Message-ID: <009101d25bee$1109e190$27b23dae@sambora>

I'm not a beginning python coder, but I'm not an advanced one either. I
can't see why I have this problem, though at this point I've probably
been looking at it too hard and for too long (several days), so maybe
I'm just too close to it.
Can one of you guys see the problem (besides my childish coding)? I'll
give you the code first, and then the problem.

def moving():
    import csv
    ls = []
    with open('E:\\Coding projects\\Pycharm\\Moving\\New Listings.csv',
'r') as infile:
        raw = csv.reader(infile)
        indata = list(raw)
        rows = indata.__len__()
    for i in range(rows):
        ls.append(indata[i])
    # sort: Description only, to make hyperelinks & find duplicates
    mergeSort(ls)
    # find & mark dups, make hyperlink if not dup
    for i in range(1, len(ls) - 1):
        if ls[i][0] == ls[i + 1][0]:
            ls[i][1] = "dup"
        else:
            # make hyperlink
            desc = ls[i][0]
            url = ls[i][1]
            ls[i][0] = '=HYPERLINK(\"' + url + '\",\"' + desc + '\")'
    # save to csv
    ls.insert(0, ["Description","url"])
    with open('E:\\Coding projects\\Pycharm\\Moving\\Moving 2017
out.csv', 'w') as outfile:
        writer = csv.writer(outfile, lineterminator='\n')
        writer.writerows(ls)

import operator
def mergeSort(L, compare = operator.lt):
    if len(L) < 2:
        return L[:]
    else:
        middle = int(len(L)/2)
        left = mergeSort(L[:middle], compare)
        right = mergeSort(L[middle:], compare)
        return merge(left, right, compare)

def merge(left, right, compare):
    result = []
    i,j = 0, 0
    while i < len(left) and j < len(right):
        if compare(left[i], right[j]):
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            j += 1
    while (i < len(left)):
        result.append(left[i])
        i += 1
    while (j < len(right)):
        result.append(right[j])
        j += 1
    return result

moving()

The problem is that while mergeSort puts the list ls in perfect order,
which I can see by looking at result on merge's final return to
mergeSort, and at the left and the right once back in mergeSort. Both
the left half and the right half are in order. But the list L is still
in its original order, and after mergeSort completes, ls is still in its
original order. Maybe there's some bonehead error causing this, but I
just can't see it.

I can provide a sample csv file for input, if you want to execute this,
but to keep things simple, you can see the problem in just a table with
webpage titles in one column and their urls in the second column.

Any insights would be greatly appreciated.



From rosuav at gmail.com  Wed Dec 21 20:46:02 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 22 Dec 2016 12:46:02 +1100
Subject: Mergesort problem
In-Reply-To: <009101d25bee$1109e190$27b23dae@sambora>
References: <36827d43-9c4e-4916-9dd1-9ffb1acd06c9@googlegroups.com>
 <009101d25bee$1109e190$27b23dae@sambora>
Message-ID: <CAPTjJmroRggeYjUamvmV8ZVfNULjdFnZ8VgiXhHwngKLOWo-yg@mail.gmail.com>

On Thu, Dec 22, 2016 at 11:55 AM, Deborah Swanson
<python at deborahswanson.net> wrote:
> The problem is that while mergeSort puts the list ls in perfect order,
> which I can see by looking at result on merge's final return to
> mergeSort, and at the left and the right once back in mergeSort. Both
> the left half and the right half are in order. But the list L is still
> in its original order, and after mergeSort completes, ls is still in its
> original order. Maybe there's some bonehead error causing this, but I
> just can't see it.
>

Your analysis is excellent. Here's what happens: When you merge-sort,
you're always returning a new list (either "return L[:]" or "result =
[]"), but then you call it like this:

    # sort: Description only, to make hyperelinks & find duplicates
    mergeSort(ls)

This calls mergeSort, then drops the newly-sorted list on the floor.
Instead, try: "ls = mergeSort(ls)".

Thank you for making it so easy for us!

ChrisA


From python at deborahswanson.net  Wed Dec 21 21:27:46 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Wed, 21 Dec 2016 18:27:46 -0800
Subject: Mergesort problem
In-Reply-To: <CAPTjJmroRggeYjUamvmV8ZVfNULjdFnZ8VgiXhHwngKLOWo-yg@mail.gmail.com>
Message-ID: <000001d25bfa$fbb3c560$27b23dae@sambora>

> On Thu, Dec 22, 2016 at 11:55 AM, Deborah Swanson 
> <python at deborahswanson.net> wrote:
> > The problem is that while mergeSort puts the list ls in 
> perfect order, 
> > which I can see by looking at result on merge's final return to 
> > mergeSort, and at the left and the right once back in 
> mergeSort. Both 
> > the left half and the right half are in order. But the list 
> L is still 
> > in its original order, and after mergeSort completes, ls is 
> still in 
> > its original order. Maybe there's some bonehead error causing this, 
> > but I just can't see it.
> >
> 
> Your analysis is excellent. Here's what happens: When you 
> merge-sort, you're always returning a new list (either 
> "return L[:]" or "result = []"), but then you call it like this:
> 
>     # sort: Description only, to make hyperelinks & find duplicates
>     mergeSort(ls)
> 
> This calls mergeSort, then drops the newly-sorted list on the 
> floor. Instead, try: "ls = mergeSort(ls)".
> 
> Thank you for making it so easy for us!
> 
> ChrisA

"ls = mergeSort(ls)" works perfectly!

I can see why now, but I'm not sure how long I would have knocked my
head against it before I saw it on my own. It must take awhile to
develop an eye for these things.

So thank you from the bottom of my heart! I do have a future in python
coding planned, but right now I need to find the cheapest nice little
house to move to, and this sorting problem was a major roadblock! The
webpage titles and urls are from Craigslist, soon to be joined by many
other fields, but I just couldn't get past this one problem.



From frank at chagford.com  Thu Dec 22 04:39:14 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 22 Dec 2016 11:39:14 +0200
Subject: [OT] Security question
Message-ID: <o3g700$vm2$1@blaine.gmane.org>

Hi all

This is off-topic, but I would appreciate a comment on this matter.

I have just upgraded my internet connection from ADSL to Fibre.

As part of the process, my ISP sent a text message to my cell phone with the 
username and password I must use to connect.

To my surprise, they sent me my existing username *and* my existing 
password, all in clear text.

I felt that this was insecure, so I sent them an email querying this and 
querying why they had my password in clear text on their system in the first 
place.

This was their reply -

"""
Thank you for taking the time to contact [...] Technical Mail Support.
I understand the importance of your password inquiry and will gladly assist.
Please note our Password protocols are secured via OTP.
This means nobody else can register or request your password as it will only 
be sent to the cellphone number we have registered for the OTP service on 
our side.
If somebody else requests a reminder of the password, it will be sent to 
your cellphone as your number is registered for the OTP service.
I hope this clarifies the matter.
"""

They did not comment on the second part of my query.

Does their reply sound reasonable, or are my concerns valid?

Thanks

Frank Millman




From rosuav at gmail.com  Thu Dec 22 04:49:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 22 Dec 2016 20:49:46 +1100
Subject: [OT] Security question
In-Reply-To: <o3g700$vm2$1@blaine.gmane.org>
References: <o3g700$vm2$1@blaine.gmane.org>
Message-ID: <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>

On Thu, Dec 22, 2016 at 8:39 PM, Frank Millman <frank at chagford.com> wrote:
> To my surprise, they sent me my existing username *and* my existing
> password, all in clear text.
>
> """
> Thank you for taking the time to contact [...] Technical Mail Support.
> I understand the importance of your password inquiry and will gladly assist.
> Please note our Password protocols are secured via OTP.
> This means nobody else can register or request your password as it will only
> be sent to the cellphone number we have registered for the OTP service on
> our side.
> If somebody else requests a reminder of the password, it will be sent to
> your cellphone as your number is registered for the OTP service.
> I hope this clarifies the matter.
> """
>
> They did not comment on the second part of my query.
>
> Does their reply sound reasonable, or are my concerns valid?

Your concerns are entirely valid. Somehow, the information of your
password got sent to you, which means that anyone who can "reach in"
at some point between where it's stored and where it's sent can leech
everyone's passwords. Game over.

If they were sending you a *new* password ("we have generated this
password, please log in and change it"), then it would be entirely
acceptable - a mobile phone text message is a decent out-of-band way
to deliver that kind of information. But to have your existing
password? No sir, no thank you, I will have none of that.

Name and shame the ISP. This kind of thing is insidious (because
usually nobody will know until it's way, WAY too late) and extremely
dangerous. Call them out on it.

ChrisA


From frank at chagford.com  Thu Dec 22 05:10:40 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 22 Dec 2016 12:10:40 +0200
Subject: [OT] Security question
In-Reply-To: <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
Message-ID: <o3g8qu$7pg$1@blaine.gmane.org>

"Chris Angelico"  wrote in message 
news:CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A at mail.gmail.com...
>
> On Thu, Dec 22, 2016 at 8:39 PM, Frank Millman <frank at chagford.com> wrote:
> > To my surprise, they sent me my existing username *and* my existing
> > password, all in clear text.
> >
>
> Your concerns are entirely valid. Somehow, the information of your
> password got sent to you, which means that anyone who can "reach in"
> at some point between where it's stored and where it's sent can leech
> everyone's passwords. Game over.
>
> If they were sending you a *new* password ("we have generated this
> password, please log in and change it"), then it would be entirely
> acceptable - a mobile phone text message is a decent out-of-band way
> to deliver that kind of information. But to have your existing
> password? No sir, no thank you, I will have none of that.
>
> Name and shame the ISP. This kind of thing is insidious (because
> usually nobody will know until it's way, WAY too late) and extremely
> dangerous. Call them out on it.
>

Thanks, Chris, good to know I am not going mad!

What about the second part of my query? Is it acceptable that they keep 
passwords on their system in clear text?

>From my first encounter with Unix over 30 years ago I was impressed with the 
fact that no passwords are stored in clear text. Even with my own little 
accounting system, I only store the SHA-1 hash of the password. I cannot 
imagine why anyone would think that this is a good idea.

The ISP is MWEB, one of the biggest service providers in South Africa, with 
(I guess) millions of users.

If this is the standard of security out there, it is no wonder we hear of so 
many attacks (and how many don't we hear of?)

Frank




From rosuav at gmail.com  Thu Dec 22 05:33:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 22 Dec 2016 21:33:52 +1100
Subject: [OT] Security question
In-Reply-To: <o3g8qu$7pg$1@blaine.gmane.org>
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
Message-ID: <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>

On Thu, Dec 22, 2016 at 9:10 PM, Frank Millman <frank at chagford.com> wrote:
> What about the second part of my query? Is it acceptable that they keep
> passwords on their system in clear text?

Well no, absolutely not. I referred to "decrypting" the password,
which is all you can actually be certain of here - they may well be
storing the passwords in an encrypted form, but it can be decrypted.

> From my first encounter with Unix over 30 years ago I was impressed with the
> fact that no passwords are stored in clear text. Even with my own little
> accounting system, I only store the SHA-1 hash of the password. I cannot
> imagine why anyone would think that this is a good idea.

>From worst to best, here's some ways you can store passwords:

1) Clear text. If anyone even just glances at your database, it's game
over in one shot.
2) Reversibly encrypted. If someone gets your database, s/he can
decrypt the contents, but accidentally flashing a long string of
nonsense won't instantly reveal everything.
3) Encrypted with a key. To decode all the passwords, you would need
additional information. That might come from the code, or from
environment variables, or something, but you would need to attack
something other than the database to completely decrypt people's
passwords.
4) Unsalted hashes. Instead of storing "password", you store
"5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8". Can be broken with rainbow
tables (or for common passwords, just Google the thing).
5) Hashes salted with something predictable or calculable. Maybe you
hash username+"blargh"+password, or something. Means the hashes don't
look the same even for the same password.
6) Hashes salted with arbitrary data that then gets stored alongside
the password.

Any of the first three could give the phenomenon you describe. And
while the security is better on #3, it's still entirely vulnerable to
the "disgruntled employee" attack (someone on the inside with complete
information about the system).

The last three all look similar in the database, but #4 is vulnerable
to XKCD 1286 attacks, and even #5 has its vulnerabilities (for
instance, setting your password to the same thing as it's previously
been will set the hash to the same value). I would recommend the use
of #6, but if someone's using #5, I wouldn't hate on them too hard -
that's pretty secure.

> The ISP is MWEB, one of the biggest service providers in South Africa, with
> (I guess) millions of users.

Thanks. MWEB, listen up: you are betraying your users' trust. A data
breach could cost you dearly. Act *now*, before there is actually a
breach, and go and hash all your passwords. And, of course, change
your systems so you never need to send people their passwords.

> If this is the standard of security out there, it is no wonder we hear of so
> many attacks (and how many don't we hear of?)

There are definitely a lot of nasty attacks out there, but these days,
the most dangerous attacks are the social ones. It doesn't matter how
good your password policy is if people will click on links in spam
email and type their passwords into imitation login screens. It
doesn't matter how well you salt your passwords if people write them
down in insecure scribble pads. It makes no difference what you do on
the back end if your users sign up for new services and use the same
email address and password on them all. But here's the thing: social
attacks are under the control of the individual user; database attacks
are under the control of the central service. MWEB is responsible for
what they do with their users' passwords, even if some of those users
are vulnerable elsewhere.

ChrisA


From frank at chagford.com  Thu Dec 22 06:10:04 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 22 Dec 2016 13:10:04 +0200
Subject: [OT] Security question
In-Reply-To: <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
 <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>
Message-ID: <o3gcaa$o6j$1@blaine.gmane.org>

"Chris Angelico"  wrote in message 
news:CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w at mail.gmail.com...
>
> On Thu, Dec 22, 2016 at 9:10 PM, Frank Millman <frank at chagford.com> wrote:
> > What about the second part of my query? Is it acceptable that they keep
> > passwords on their system in clear text?
>
> Well no, absolutely not. I referred to "decrypting" the password,
> which is all you can actually be certain of here - they may well be
> storing the passwords in an encrypted form, but it can be decrypted.
>

[snip much fascinating info]

Thanks for all the info, Chris.

This is clearly a subject you feel strongly about!

Much appreciated.

Frank




From rosuav at gmail.com  Thu Dec 22 06:17:35 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 22 Dec 2016 22:17:35 +1100
Subject: [OT] Security question
In-Reply-To: <o3gcaa$o6j$1@blaine.gmane.org>
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
 <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>
 <o3gcaa$o6j$1@blaine.gmane.org>
Message-ID: <CAPTjJmoFrS85FqiN=W6P1Xcmnvd+RBgDPrVNKW-EfbUbgAcTiw@mail.gmail.com>

On Thu, Dec 22, 2016 at 10:10 PM, Frank Millman <frank at chagford.com> wrote:
> Thanks for all the info, Chris.
>
> This is clearly a subject you feel strongly about!
>
> Much appreciated.

It is - partly because I've been guilty of poor password security in
the past. I speak with the voice of someone who has sighted horrors in
his future and escaped by sheer luck (nobody actually leeched my
database... that I know of, at least). My systems were much smaller
and less serious than an ISP's.

ChrisA


From robin at reportlab.com  Thu Dec 22 08:43:46 2016
From: robin at reportlab.com (Robin Becker)
Date: Thu, 22 Dec 2016 13:43:46 +0000
Subject: htmlparser charrefs
Message-ID: <8160d131-fbd1-5188-1cd3-642d8cfb6636@chamonix.reportlab.co.uk>

For various reasons I am using HTMLParser to do transcription of some xml. I 
need to keep charrefs as is so for Python > 3.4 I pass in

convert_charrefs =False

to the constructor.

This seems to work as intended for data, but I notice that a change in Python 
3.4 prevents me from keeping the charrefs which are in attribute strings.

Is it intentional that we can no longer use HTMLParser.unescape? It seems to 
prevent correct interpretation of the convert_charrefs constructor argument.

The unescaping is done, but in a module level function which means I can no 
longer override that functionality safely.
-- 
Robin Becker



From and.damore at gmail.com  Thu Dec 22 08:58:35 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Thu, 22 Dec 2016 14:58:35 +0100
Subject: Mergesort problem
In-Reply-To: <009101d25bee$1109e190$27b23dae@sambora>
References: <36827d43-9c4e-4916-9dd1-9ffb1acd06c9@googlegroups.com>
 <009101d25bee$1109e190$27b23dae@sambora>
Message-ID: <CAMonFrSx+nZTb5nw36KD8taz-xbDfFiA3h88jqZ6VVLRZ2akEg@mail.gmail.com>

I know a code review wasn't the main goal of you message but I feel
it's worth mentioning two tips:

On 22 December 2016 at 01:55, Deborah Swanson <python at deborahswanson.net> wrote:
>     ls = []
>     with open('E:\\Coding projects\\Pycharm\\Moving\\New Listings.csv',
> 'r') as infile:
>         raw = csv.reader(infile)
>         indata = list(raw)
>         rows = indata.__len__()
>     for i in range(rows):
>         ls.append(indata[i])

This block init an empty list, creates a csv.reader, processes it all
converting to a list then loops over every in this list and assign
this item to the initially created list. The initial list object is
actually useless since at the end ls and rows will contain exactly the
same objects.
Your code can be simplified with:

     with open(your_file_path) as infile:
         ls = list(csv.reader(infile))


Then I see you're looping with an index-based approach, here

>     for i in range(rows):
>         ls.append(indata[i])
[?]
>     # find & mark dups, make hyperlink if not dup
>     for i in range(1, len(ls) - 1):

and in the other functions, basically wherever you use len().

Check Ned Batchelders's "Loop like a native" talk about that, there
are both a webpage and a PyCon talk.
By using "native" looping you'll get simplified code that is more
expressive in less lines.


-- 
Andrea


From lists at ozindfw.net  Thu Dec 22 09:12:04 2016
From: lists at ozindfw.net (Rich Osman)
Date: Thu, 22 Dec 2016 08:12:04 -0600
Subject: [OT] Security question
In-Reply-To: <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
 <CAPTjJmrG+1==NmOXF6CU2PttgCYKGZ_Dvi36GjaqHqA9daFx5w@mail.gmail.com>
Message-ID: <B6CED319-06FB-4895-A600-F3404F69CD97@ozindfw.net>

Chris,

I compliment you on your succint and accurate summary of the issue.

Sounds like Frank's ISP is aspiring to be the next Yahoo...

On December 22, 2016 4:33:52 AM CST, Chris Angelico <rosuav at gmail.com> wrote:
>On Thu, Dec 22, 2016 at 9:10 PM, Frank Millman <frank at chagford.com>
>wrote:
>> What about the second part of my query? Is it acceptable that they
>keep
>> passwords on their system in clear text?
>
>Well no, absolutely not. I referred to "decrypting" the password,
>which is all you can actually be certain of here - they may well be
>storing the passwords in an encrypted form, but it can be decrypted.
>
>> From my first encounter with Unix over 30 years ago I was impressed
>with the
>> fact that no passwords are stored in clear text. Even with my own
>little
>> accounting system, I only store the SHA-1 hash of the password. I
>cannot
>> imagine why anyone would think that this is a good idea.
>
>From worst to best, here's some ways you can store passwords:
>
>1) Clear text. If anyone even just glances at your database, it's game
>over in one shot.
>2) Reversibly encrypted. If someone gets your database, s/he can
>decrypt the contents, but accidentally flashing a long string of
>nonsense won't instantly reveal everything.
>3) Encrypted with a key. To decode all the passwords, you would need
>additional information. That might come from the code, or from
>environment variables, or something, but you would need to attack
>something other than the database to completely decrypt people's
>passwords.
>4) Unsalted hashes. Instead of storing "password", you store
>"5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8". Can be broken with rainbow
>tables (or for common passwords, just Google the thing).
>5) Hashes salted with something predictable or calculable. Maybe you
>hash username+"blargh"+password, or something. Means the hashes don't
>look the same even for the same password.
>6) Hashes salted with arbitrary data that then gets stored alongside
>the password.
>
>Any of the first three could give the phenomenon you describe. And
>while the security is better on #3, it's still entirely vulnerable to
>the "disgruntled employee" attack (someone on the inside with complete
>information about the system).
>
>The last three all look similar in the database, but #4 is vulnerable
>to XKCD 1286 attacks, and even #5 has its vulnerabilities (for
>instance, setting your password to the same thing as it's previously
>been will set the hash to the same value). I would recommend the use
>of #6, but if someone's using #5, I wouldn't hate on them too hard -
>that's pretty secure.
>
>> The ISP is MWEB, one of the biggest service providers in South
>Africa, with
>> (I guess) millions of users.
>
>Thanks. MWEB, listen up: you are betraying your users' trust. A data
>breach could cost you dearly. Act *now*, before there is actually a
>breach, and go and hash all your passwords. And, of course, change
>your systems so you never need to send people their passwords.
>
>> If this is the standard of security out there, it is no wonder we
>hear of so
>> many attacks (and how many don't we hear of?)
>
>There are definitely a lot of nasty attacks out there, but these days,
>the most dangerous attacks are the social ones. It doesn't matter how
>good your password policy is if people will click on links in spam
>email and type their passwords into imitation login screens. It
>doesn't matter how well you salt your passwords if people write them
>down in insecure scribble pads. It makes no difference what you do on
>the back end if your users sign up for new services and use the same
>email address and password on them all. But here's the thing: social
>attacks are under the control of the individual user; database attacks
>are under the control of the central service. MWEB is responsible for
>what they do with their users' passwords, even if some of those users
>are vulnerable elsewhere.
>
>ChrisA
>-- 
>https://mail.python.org/mailman/listinfo/python-list

-- 
mailto:oz at ozindfw.net    
Oz
POB 93167 
Southlake, TX 76092 (Near DFW Airport

From uri at speedy.net  Thu Dec 22 09:12:17 2016
From: uri at speedy.net (Uri Even-Chen)
Date: Thu, 22 Dec 2016 16:12:17 +0200
Subject: ImportError: The ``fake-factory`` package is now called ``Faker``.
In-Reply-To: <E315C06B-B4E6-4A2A-98C2-0708B839813E@williammayor.co.uk>
References: <CAMQ2MsEjOtyUryABSr2qnN17C-B=jZM2szoR1W-fVGQxmBB4fQ@mail.gmail.com>
 <E315C06B-B4E6-4A2A-98C2-0708B839813E@williammayor.co.uk>
Message-ID: <CAMQ2MsHbDpx3+Y5OBTYR4zC4sh826rRA4qx0G595UJNo4dWAqA@mail.gmail.com>

Thank you, I found out that pip freeze returned more packages than what we
had in our requirements.txt file, so I upgraded them and added them to the
file. I also upgraded to Faker==0.7.7

Uri.

*Uri Even-Chen*
[image: photo] Phone: +972-54-3995700
Email: uri at speedy.net
Website: http://www.speedysoftware.com/uri/en/
<http://www.facebook.com/urievenchen>  <http://plus.google.com/+urievenchen>
  <http://www.linkedin.com/in/urievenchen>  <http://github.com/urievenchen>
<http://twitter.com/urievenchen>

On Tue, Dec 20, 2016 at 7:11 PM, William Mayor <mail at williammayor.co.uk>
wrote:

> I came across this error this morning.
>
> In my case I was running a script that did this: ?pip install fake-factory?
>
> The install worked properly but then I couldn?t import the library (with
> the same error message as you).
>
> I had to update the pip install command to say ?pip install Faker? and
> then everything worked as before.
>
> The authors appear to have forced a name change: https://github.com/
> joke2k/faker/blob/fake-factory/faker/__init__.py
>
> I?ve just checked and if you pin your requirements (e.g. pip install
> fake-factory==0.7.4) then it still works.
>
>
> On 20 Dec 2016, at 16:57, Uri Even-Chen <uri at speedy.net> wrote:
>
> Hi, we get this error with Python 3.4 and 3.5:
> https://travis-ci.org/urievenchen/speedy-net/jobs/185497863
>
>
> ----------------------------------------------------------------------
>
> ImportError: Failed to import test module:
> speedy.net.accounts.tests.test_factories
>
> Traceback (most recent call last):
>
>  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 428,
> in _find_test_path
>
>    module = self._get_module_from_name(name)
>
>  File "/opt/python/3.5.2/lib/python3.5/unittest/loader.py", line 369,
> in _get_module_from_name
>
>    __import__(name)
>
>  File "/home/travis/build/urievenchen/speedy-net/speedy/
> net/accounts/tests/test_factories.py",
> line 4, in <module>
>
>    import factory
>
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/
> site-packages/factory/__init__.py",
> line 46, in <module>
>
>    from .faker import Faker
>
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/
> site-packages/factory/faker.py",
> line 41, in <module>
>
>    import faker
>
>  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/
> site-packages/faker/__init__.py",
> line 7, in <module>
>
>    raise ImportError(error)
>
> ImportError: The ``fake-factory`` package is now called ``Faker``.
>
> Please update your requirements.
>
>
> What is the problem? We didn't update our requirements recently.
>
> Thanks,
> Uri.
>
> *Uri Even-Chen*
> [image: photo] Phone: +972-54-3995700 <+972%2054-399-5700>
> Email: uri at speedy.net
> Website: http://www.speedysoftware.com/uri/en/
> <http://www.facebook.com/urievenchen>  <http://plus.google.com/+
> urievenchen>
>  <http://www.linkedin.com/in/urievenchen>  <http://github.com/urievenchen>
> <http://twitter.com/urievenchen>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
>


From info at wingware.com  Thu Dec 22 10:00:57 2016
From: info at wingware.com (Wingware)
Date: Thu, 22 Dec 2016 10:00:57 -0500
Subject: ANN:  Wing IDE 6.0 has been released
Message-ID: <585BEAA9.404@wingware.com>

Hi,

We've just released Wing IDE 6.0, which is a major release that adds 
many new features, introduces a new annual license option, and makes 
some changes to the product line.

New Features

  * Improved Multiple Selections:  Quickly add selections and edit them
    all at once
  * Easy Remote Development:  Work seamlessly on remote Linux, OS X, and
    Raspberry Pi systems
  * Debugging in the Python Shell:  Reach breakpoints and exceptions in
    (and from) the Python Shell
  * Recursive Debugging:  Debug code invoked in the context of stack
    frames that are already being debugged
  * PEP 484 and PEP 526 Type Hinting:  Inform Wing's static analysis
    engine of types it cannot infer
  * Support for Python 3.6 and Stackless 3.4:  Use async and other new
    language features
  * Optimized debugger:  Run faster, particularly in multi-process and
    multi-threaded code
  * Support for OS X full screen mode:  Zoom to a virtual screen, with
    auto-hiding menu bar
  * Added a new One Dark color palette:  Enjoy the best dark display
    style yet
  * Updated French and German localizations:  Thanks to Jean Sanchez,
    Laurent Fasnacht, and Christoph Heitkamp

For a much more detailed overview of new features see the release notice 
at http://wingware.com/news/2016-12-20

Annual Use License Option

Wing 6 adds the option of purchasing a lower-cost expiring annual 
license for Wing IDE Pro.  An annual license includes access to all 
available Wing IDE versions while it is valid, and then ceases to 
function if it is now renewed.  Pricing for annual licenses is US$ 
179/user for Commercial Use and US$ 69/user for Non-Commercial Use.

Perpetual licenses for Wing IDE will continue to be available at the 
same pricing.

The cost of extending Support+Upgrades subscriptions on Non-Commercial 
Use perpetual licenses for Wing IDE Pro has also been dropped from US$ 
89 to US$ 39 per user.

For details, see https://wingware.com/store/purchase

Wing Personal is Free

Wing IDE Personal is now free and no longer requires a license to run. 
  It now also includes the Source Browser, PyLint, and OS Commands 
tools, and supports the scripting API and Perspectives.

However, Wing Personal does not include Wing Pro's advanced editing, 
debugging, testing and code management features, such as remote host 
access, refactoring, find uses, version control, unit testing, 
interactive debug probe, multi-process and child process debugging, move 
program counter, conditional breakpoints, debug watch, 
framework-specific support (for matplotlib, Django, and others), find 
symbol in project, and other features.

Links

Release notice: http://wingware.com/news/2016-12-20
Free trial: http://wingware.com/wingide/trial
Downloads: http://wingware.com/downloads
Compare products: http://wingware.com/wingide/features
Buy: http://wingware.com/store/purchase
Upgrade: https://wingware.com/store/upgrade

Questions?  Don't hesitate to email us at support at wingware.com.

Thanks,

-- 

Stephan Deibel
Wingware | Python IDE

The Intelligent Development Environment for Python Programmers

wingware.com



From skip.montanaro at gmail.com  Thu Dec 22 15:49:28 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 22 Dec 2016 14:49:28 -0600
Subject: US/Eastern offset
Message-ID: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>

In a small application I realized I needed all my timestamps to have
timezone info. Some timestamp strings come in with no TZ markings, but
I know they are US/Eastern. so, I built one:

>>> import pytz
>>> tz = pytz.timezone("US/Eastern")
>>> tz
<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>

What's with those extra four minutes? Here is one such timestamp I
logged in my app:

    2016-12-22T20:35:05-04:56

WTF? Has my brain turned to mush, and the people in New York now move
so fast that they are four minutes closer to their London counterparts
than they used to be?

Thx,

Skip


From ian.g.kelly at gmail.com  Thu Dec 22 17:19:08 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Thu, 22 Dec 2016 16:19:08 -0600
Subject: US/Eastern offset
In-Reply-To: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
Message-ID: <CALwzidnr+e2Xg6C5WE=ot1hZN8MO4OJs7WOGoreZB1o2gvbmaA@mail.gmail.com>

On Thu, Dec 22, 2016 at 2:49 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> In a small application I realized I needed all my timestamps to have
> timezone info. Some timestamp strings come in with no TZ markings, but
> I know they are US/Eastern. so, I built one:
>
>>>> import pytz
>>>> tz = pytz.timezone("US/Eastern")
>>>> tz
> <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
>
> What's with those extra four minutes? Here is one such timestamp I
> logged in my app:
>
>     2016-12-22T20:35:05-04:56
>
> WTF? Has my brain turned to mush, and the people in New York now move
> so fast that they are four minutes closer to their London counterparts
> than they used to be?

Not sure, but LMT suggests that it's coming up with a Local Mean Time
zone rather than the proper EST zone. The four minute offset suggests
that this is a local mean time for a meridian one degree east of EST.
I'm wondering if it's using some historical definition for the time
zone's date-neutral repr. What do you get if you try using it to
convert a UTC time? E.g.:

py> dt = pytz.utc.localize(datetime(2016, 12, 23))
py> dt.astimezone(pytz.timezone('US/Eastern'))
datetime.datetime(2016, 12, 22, 19, 0, tzinfo=<DstTzInfo 'US/Eastern'
EST-1 day, 19:00:00 STD>)


From ian.g.kelly at gmail.com  Thu Dec 22 18:05:28 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Thu, 22 Dec 2016 17:05:28 -0600
Subject: US/Eastern offset
In-Reply-To: <CALwzidnr+e2Xg6C5WE=ot1hZN8MO4OJs7WOGoreZB1o2gvbmaA@mail.gmail.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <CALwzidnr+e2Xg6C5WE=ot1hZN8MO4OJs7WOGoreZB1o2gvbmaA@mail.gmail.com>
Message-ID: <CALwzid=x2J2=yJpQH0noPQdEPC5nSE+eYoLNRkMgVDMohOMM3g@mail.gmail.com>

On Thu, Dec 22, 2016 at 4:19 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> On Thu, Dec 22, 2016 at 2:49 PM, Skip Montanaro
> <skip.montanaro at gmail.com> wrote:
>> In a small application I realized I needed all my timestamps to have
>> timezone info. Some timestamp strings come in with no TZ markings, but
>> I know they are US/Eastern. so, I built one:
>>
>>>>> import pytz
>>>>> tz = pytz.timezone("US/Eastern")
>>>>> tz
>> <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
>>
>> What's with those extra four minutes? Here is one such timestamp I
>> logged in my app:
>>
>>     2016-12-22T20:35:05-04:56
>>
>> WTF? Has my brain turned to mush, and the people in New York now move
>> so fast that they are four minutes closer to their London counterparts
>> than they used to be?
>
> Not sure, but LMT suggests that it's coming up with a Local Mean Time
> zone rather than the proper EST zone. The four minute offset suggests
> that this is a local mean time for a meridian one degree east of EST.

You mentioned New York in your post, and incidentally the 74th
meridian west passes right through New York City. So it could be an
LMT for NYC, though I'm still not sure why it would come up with that
specifically for US/Eastern.


From christian at python.org  Thu Dec 22 18:46:33 2016
From: christian at python.org (Christian Heimes)
Date: Fri, 23 Dec 2016 00:46:33 +0100
Subject: US/Eastern offset
In-Reply-To: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
Message-ID: <o3hokk$cl8$1@blaine.gmane.org>

On 2016-12-22 21:49, Skip Montanaro wrote:
> In a small application I realized I needed all my timestamps to have
> timezone info. Some timestamp strings come in with no TZ markings, but
> I know they are US/Eastern. so, I built one:
> 
>>>> import pytz
>>>> tz = pytz.timezone("US/Eastern")
>>>> tz
> <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
> 
> What's with those extra four minutes? Here is one such timestamp I
> logged in my app:
> 
>     2016-12-22T20:35:05-04:56
> 
> WTF? Has my brain turned to mush, and the people in New York now move
> so fast that they are four minutes closer to their London counterparts
> than they used to be?

pytz contains not only current time zone, but also historic time zones.
You are looking at a time zone without a date and time context. Without
a context, pytz shows you a historic time zone information. In your case
pytz gives you the local mean time (solar time) the 19th century.

Christian





From churros.studios at gmail.com  Thu Dec 22 20:32:03 2016
From: churros.studios at gmail.com (churros.studios at gmail.com)
Date: Thu, 22 Dec 2016 17:32:03 -0800 (PST)
Subject: SIP and PyQt5 Installation
Message-ID: <327c1b0a-bb8c-4c5e-b1cc-9825d9446f22@googlegroups.com>

Hello, I've been having troubles being able to integrate the PyQt5 GUI into my IDEs. I've used various example codes from the web to see if the library would import properly but the console sends me back an import error saying, "No module named PyQt5". Could someone provide me with the steps for not only the installation/configuring of SIP/PyQt but also the steps needed to integrate it to the eclipse IDE?


From jladasky at itu.edu  Thu Dec 22 20:54:32 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Thu, 22 Dec 2016 17:54:32 -0800 (PST)
Subject: US/Eastern offset
In-Reply-To: <mailman.216.1482450413.2322.python-list@python.org>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <o3hokk$cl8$1@blaine.gmane.org>
 <mailman.216.1482450413.2322.python-list@python.org>
Message-ID: <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>

On Thursday, December 22, 2016 at 3:47:04 PM UTC-8, Christian Heimes wrote:
> On 2016-12-22 21:49, Skip Montanaro wrote:
> > In a small application I realized I needed all my timestamps to have
> > timezone info. Some timestamp strings come in with no TZ markings, but
> > I know they are US/Eastern. so, I built one:
> > 
> >>>> import pytz
> >>>> tz = pytz.timezone("US/Eastern")
> >>>> tz
> > <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
> > 
> > What's with those extra four minutes? Here is one such timestamp I
> > logged in my app:
> > 
> >     2016-12-22T20:35:05-04:56
> > 
> > WTF? Has my brain turned to mush, and the people in New York now move
> > so fast that they are four minutes closer to their London counterparts
> > than they used to be?
> 
> pytz contains not only current time zone, but also historic time zones.
> You are looking at a time zone without a date and time context. Without
> a context, pytz shows you a historic time zone information. In your case
> pytz gives you the local mean time (solar time) the 19th century.
> 
> Christian

Wouldn't most users prefer that modern time zones be the default information returned by pytz, instead of 150 year-old historical time zones?


From rosuav at gmail.com  Thu Dec 22 20:57:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 23 Dec 2016 12:57:28 +1100
Subject: US/Eastern offset
In-Reply-To: <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <o3hokk$cl8$1@blaine.gmane.org>
 <mailman.216.1482450413.2322.python-list@python.org>
 <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>
Message-ID: <CAPTjJmqtJq8yfQEZ=jFWs3ApA+485dexFEcRBXado=2oa6-aGg@mail.gmail.com>

On Fri, Dec 23, 2016 at 12:54 PM,  <jladasky at itu.edu> wrote:
> Wouldn't most users prefer that modern time zones be the default information returned by pytz, instead of 150 year-old historical time zones?

They're the *same* time zones.

ChrisA


From jladasky at itu.edu  Thu Dec 22 21:00:17 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Thu, 22 Dec 2016 18:00:17 -0800 (PST)
Subject: SIP and PyQt5 Installation
In-Reply-To: <327c1b0a-bb8c-4c5e-b1cc-9825d9446f22@googlegroups.com>
References: <327c1b0a-bb8c-4c5e-b1cc-9825d9446f22@googlegroups.com>
Message-ID: <25d9321d-3629-41a4-ba33-9c29b4384e20@googlegroups.com>

On Thursday, December 22, 2016 at 5:32:31 PM UTC-8, churros... at gmail.com wrote:
> Hello, I've been having troubles being able to integrate the PyQt5 GUI into my IDEs. I've used various example codes from the web to see if the library would import properly but the console sends me back an import error saying, "No module named PyQt5". Could someone provide me with the steps for not only the installation/configuring of SIP/PyQt but also the steps needed to integrate it to the eclipse IDE?

You need to specify your operating system.  I found installing PyQt and SIP to be quite simple on Ubuntu Linux, although I don't use Eclipse.  

You probably also should let people know whether you are using Python 2 or Python 3 (and if you're new to Python and don't have to work with legacy code, you will be better served by working in Py3).

About 18 months ago, I helped two people install PyQt and SIP on their machines, one Windows, one Mac.  I found it to be quite painful in both cases (and, sorry to say, I've forgotten the details).


From pkpearson at nowhere.invalid  Thu Dec 22 21:01:08 2016
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 23 Dec 2016 02:01:08 GMT
Subject: [OT] Security question
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
 <mailman.202.1482401461.2322.python-list@python.org>
Message-ID: <ec3er4Frsk2U1@mid.individual.net>

On Thu, 22 Dec 2016 12:10:40 +0200, Frank Millman <frank at chagford.com> wrote:
[snip]
>
> What about the second part of my query? Is it acceptable that they keep 
> passwords on their system in clear text?

Absolutely not.  Keeping the passwords, even encrypted, is a reckless
invitation to disaster.

Chris has done a fine job of explaining the situation; I'm just piling
on (as a retired cryptologist) to add statistical weight to your survey.

-- 
To email me, substitute nowhere->runbox, invalid->com.


From miragewebstudio12 at gmail.com  Thu Dec 22 21:23:51 2016
From: miragewebstudio12 at gmail.com (Mirage Web Studio)
Date: Fri, 23 Dec 2016 07:53:51 +0530
Subject: SIP and PyQt5 Installation
In-Reply-To: <327c1b0a-bb8c-4c5e-b1cc-9825d9446f22@googlegroups.com>
References: <327c1b0a-bb8c-4c5e-b1cc-9825d9446f22@googlegroups.com>
Message-ID: <C4AF10B6-B4A2-4DC9-ADC2-48A0AEC8F50C@gmail.com>



On December 23, 2016 7:02:03 AM GMT+05:30, churros.studios at gmail.com wrote:
>Hello, I've been having troubles being able to integrate the PyQt5 GUI
>into my IDEs. I've used various example codes from the web to see if
>the library would import properly but the console sends me back an
>import error saying, "No module named PyQt5". Could someone provide me
>with the steps for not only the installation/configuring of SIP/PyQt
>but also the steps needed to integrate it to the eclipse IDE?
>-- 
>https://mail.python.org/mailman/listinfo/python-list

-- 

If you're on windows and python 3.4 would be sufficient,  then for python 3.4 you could download windows installer from the pyqt site. Install it after installing  python and then you can  Import the libraries as regular libraries. 

George 

From subhabangalore at gmail.com  Fri Dec 23 01:38:15 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Thu, 22 Dec 2016 22:38:15 -0800 (PST)
Subject: UTF-8 Encoding Error
Message-ID: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>

I am getting the error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15: invalid start byte

as I try to read some files through TaggedCorpusReader. TaggedCorpusReader is a module
of NLTK.
My files are saved in ANSI format in MS-Windows default. 
I am using Python2.7 on MS-Windows 7. 

I have tried the following options till now, 
string.encode('utf-8').strip()
unicode(string)
unicode(str, errors='replace')
unicode(str, errors='ignore')
string.decode('cp1252')

But nothing is of much help.

If any one may kindly suggest.

I am trying if you may see.


From cs at zip.com.au  Fri Dec 23 01:58:15 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 23 Dec 2016 17:58:15 +1100
Subject: UTF-8 Encoding Error
In-Reply-To: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
Message-ID: <20161223065815.GA72019@cskk.homeip.net>

On 22Dec2016 22:38, Subhabrata Banerjee <subhabangalore at gmail.com> wrote:
>I am getting the error:
>UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15: invalid start byte
>
>as I try to read some files through TaggedCorpusReader. TaggedCorpusReader is a module
>of NLTK.
>My files are saved in ANSI format in MS-Windows default.
>I am using Python2.7 on MS-Windows 7.
>
>I have tried the following options till now,
>string.encode('utf-8').strip()
>unicode(string)
>unicode(str, errors='replace')
>unicode(str, errors='ignore')
>string.decode('cp1252')
>
>But nothing is of much help.

It would help to see a very small program that produces your error message.

Generally you need to open text files in the same encoding used for thei text.  
Which sounds obvious, but I'm presuming you've not done that.

Normally, when you open a file you can specify the text encoding. I am not a 
Windows guy, so I do not know what "ANSI format in MS-Windows default" means at 
the encoding level.

Supposing you had a bit of code like this:

  with open("filename.txt", "r") as fp:
      for line in fp:
          # line is a Python 2 str, but is a sequence of bytes internally
          unicode_line = line.decode('utf8')
          # unicode_line is a Python 2 _unicode_ object, which is text, a 
          # sequence of Unicode codepoints

you could get an error like yours if the file _did not_ contain UTF-8 encoded 
text.

If you used:
    unicode(str, errors='replace')
    unicode(str, errors='ignore')
I would not have expected the error you recite, but we would need to see an 
example program to be sure.

I would guess that the text in your file is not UTF-8 encoded, and that you 
need to specify the correct encoding to the .decode call.

Cheers,
Cameron Simpson <cs at zip.com.au>


From mishra15mahesh at gmail.com  Fri Dec 23 03:31:24 2016
From: mishra15mahesh at gmail.com (MAHESH MISHRA)
Date: Fri, 23 Dec 2016 00:31:24 -0800
Subject: not able to run python.exe file successsfully
Message-ID: <CAOxjLkVuZyU5xyCYxqepyHFpzc1LCtK0-iYQxbf7PnSc=CSfzg@mail.gmail.com>

i have installed 3.5.2 version of python my system windows 8.1 64 bit.
after successful installation it is not executing python.exe file.whenever
i try a dialof box pop up with an error message "python has stopped
working". i hav tried reinstalling it several times.please help


From steve+python at pearwood.info  Fri Dec 23 04:53:01 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 23 Dec 2016 20:53:01 +1100
Subject: [OT] Security question
References: <o3g700$vm2$1@blaine.gmane.org>
 <CAPTjJmoQK39EU=M3w1zr8Xa7MYv42KYN4mXPRgQMye4rGa+X4A@mail.gmail.com>
 <o3g8qu$7pg$1@blaine.gmane.org>
 <mailman.202.1482401461.2322.python-list@python.org>
Message-ID: <585cf3ff$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Thu, 22 Dec 2016 09:10 pm, Frank Millman wrote:

> If this is the standard of security out there, it is no wonder we hear of
> so many attacks (and how many don't we hear of?)

Everything is broken:

https://medium.com/message/everything-is-broken-81e5f33a24e1




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From frank at chagford.com  Fri Dec 23 05:19:00 2016
From: frank at chagford.com (Frank Millman)
Date: Fri, 23 Dec 2016 12:19:00 +0200
Subject: Another security question
Message-ID: <o3itmj$t42$1@blaine.gmane.org>

Hi all

This is a follow-up to my recent 'security question' post.

I am starting a new thread, for 2 reasons -

1) I sent a link to the previous thread to my ISP for their information. It 
is up to them whether they do anything with it, but I wanted to keep that 
thread focused on the original issue raised.

2) This one is more on-topic, as it is to do with my python project.

Having read the previous thread and various links, I want to review the way 
I handle passwords in my accounting application.

At present I just store a SHA-1 hash of the password for each user. Here are 
my thoughts on improving this.

1. Generate a 'salt' for each password. There seem to be two ways in the 
standard library to do this -
    import os
    salt = os.urandom(16)

    import secrets
    salt = secrets.token_bytes(16)

    My guess is that it will not make much difference which I use.

2. Store the salt in the database along with the user-id and hashed password 
for each user.

3. Generate the password from the string supplied by the user as follows -
    from hashlib import blake2b
    password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()

The hashlib docs have the following warning -

"Salted hashing (or just hashing) with BLAKE2 or any other general-purpose 
cryptographic hash function, such as SHA-256, is not suitable for hashing 
passwords. See BLAKE2 FAQ for more information."

I propose to ignore this warning. I feel that, for my purposes, the above 
procedure is adequate.

Does all this sound reasonable?

Any comments appreciated.

Frank Millman




From nad at python.org  Fri Dec 23 05:34:48 2016
From: nad at python.org (Ned Deily)
Date: Fri, 23 Dec 2016 05:34:48 -0500
Subject: [RELEASE] Python 3.6.0 is released!
Message-ID: <1C5335CD-4028-4F1C-A5EC-6A4DA615552E@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I am pleased to announce the availability of Python 3.6.0.  Python
3.6.0 is the newest major release of the Python language, and it contains
many new features and optimizations.  See the "What?s New In Python 3.6"
document for more information:

    https://docs.python.org/3.6/whatsnew/3.6.html

You can download Python 3.6.0 here:

    https://www.python.org/downloads/release/python-360/

Also, most third-party distributors of Python should be making 3.6.0
packages available soon.

Maintenance releases for the 3.6 series will follow at regular intervals
starting in the first quarter of 2017.

We hope you enjoy Python 3.6.0!

P.S. As a volunteer-staffed open source project, we could not bring
Python releases to you without the enormous contributions of many,
many people.  Thank you to all who have contributed and reviewed code
and documentation changes, documented and investigated bugs, tested
Python and third-party packages, and provided and supported the
infrastructure needed to support Python development and testing.
Please consider supporting the work of the Python Software Foundation.
More at:

    https://www.python.org/psf-landing/

--
  Ned Deily
  nad at python.org -- []



From steve+python at pearwood.info  Fri Dec 23 05:46:53 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 23 Dec 2016 21:46:53 +1100
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
Message-ID: <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:

[...]
> Having read the previous thread and various links, I want to review the
> way I handle passwords in my accounting application.
> 
> At present I just store a SHA-1 hash of the password for each user. Here
> are my thoughts on improving this.

SHA-1 hashes are (I believe) vulnerable to pre-computed rainbow tables. This
is where the salt comes in: the salt is not a secret itself, but if
prevents attackers pre-computing rainbow tables.

The question is, is SHA-1 (plus salting) strong enough? I'm not sure.


> 1. Generate a 'salt' for each password. There seem to be two ways in the
> standard library to do this -
>     import os
>     salt = os.urandom(16)
> 
>     import secrets
>     salt = secrets.token_bytes(16)
> 
>     My guess is that it will not make much difference which I use.

secrets is officially the preferred mechanism. os.urandom is the low-level
operating system routine, secrets.* is the high-level interface.

At the moment it is true that there's very little difference in practice,
but that's an implementation detail which could change.



> 2. Store the salt in the database along with the user-id and hashed
> password for each user.
> 
> 3. Generate the password from the string supplied by the user as follows -
>     from hashlib import blake2b
>     password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
> 
> The hashlib docs have the following warning -
> 
> "Salted hashing (or just hashing) with BLAKE2 or any other general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for hashing
> passwords. See BLAKE2 FAQ for more information."

Why are using Blake2 when the docs explicitly say not to use them in this
way? Have you read the FAQ to see what it says?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From web at nhs.net  Fri Dec 23 05:50:44 2016
From: web at nhs.net (NHS Digital - Digital Communications)
Date: Fri, 23 Dec 2016 10:50:44 +0000
Subject: PLEASE READ - information on (Case 58158) [RELEASE] Python 3.6.0
 is released!
Message-ID: <585d7578b9df4667a8772d73e2d21bca@NH-HEPEX125.AD1.NHS.NET>

Thank you for your email. Your web change 58158[RELEASE] Python 3.6.0 is released!request has been received and will be dealt with shortly.

This service desk only covers minor changes to the legacy NHS Digital website (content.digital.nhs.uk) that are made on a web request form.

For changes relating to the new beta NHS Digital website or any other communications request please refer to the intranet and complete a communications support request form.


********************************************************************************************************************

This message may contain confidential information. If you are not the intended recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents:
to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland
NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients
NHSmail provides an email address for your career in the NHS and can be accessed anywhere
For more information and to find out how you can switch, visit www.nhsdigital.nhs.uk/nhsmail

********************************************************************************************************************


From frank at chagford.com  Fri Dec 23 06:08:46 2016
From: frank at chagford.com (Frank Millman)
Date: Fri, 23 Dec 2016 13:08:46 +0200
Subject: Another security question
In-Reply-To: <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o3j0jt$3pc$1@blaine.gmane.org>

"Steve D'Aprano"  wrote in message 
news:585d009f$0$1599$c3e8da3$5496439d at news.astraweb.com...
>
> On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:
>
> >
> > 3. Generate the password from the string supplied by the user as 
> > follows -
> >     from hashlib import blake2b
> >     password = blake2b('my_password'.encode('utf-8'), 
> > salt=salt).digest()
> >
> > The hashlib docs have the following warning -
> >
> > "Salted hashing (or just hashing) with BLAKE2 or any other 
> > general-purpose
> > cryptographic hash function, such as SHA-256, is not suitable for 
> > hashing
> > passwords. See BLAKE2 FAQ for more information."
>
> Why are using Blake2 when the docs explicitly say not to use them in this
> way? Have you read the FAQ to see what it says?
>

Why am I using Blake2? Well, before today I had not heard of it. However, in 
the past, when I needed to create a hashed password, I used the built-in 
hashlib module. Today, when I look at the docs for hashlib in Python 3.6, 
this is the new sub-heading -
    "15.2. hashlib ? BLAKE2 hash functions"

So it appears that this is the new preferred way of doing it.

This is what the Blake2 FAQ says -

"You shouldn't use *any* general-purpose hash function for user passwords, 
not BLAKE2, and not MD5, SHA-1, SHA-256, or SHA-3. Instead you should use a 
password hashing function such as the PHC winner Argon2 with appropriate 
time and memory cost parameters, to mitigate the risk of bruteforce 
attacks?Argon2's core uses a variant of BLAKE2's permutation"

I see that there is a Python implementation of Argon2 in PyPi, but I don't 
particularly want to add another dependency to my app. My gut-feel says that 
this is overkill for my requirement. However, I am not sure. That is partly 
why I started this thread, to get some counter-arguments.

Frank




From mr at e-wrobel.pl  Fri Dec 23 06:14:15 2016
From: mr at e-wrobel.pl (Mr. Wrobel)
Date: Fri, 23 Dec 2016 12:14:15 +0100
Subject: Metaclasses - magic functions
In-Reply-To: <mailman.186.1482285005.2322.python-list@python.org>
References: <o3c6d8$sru$1@news.icm.edu.pl> <85a8bqxj9m.fsf@benfinney.id.au>
 <5859E013.8010108@stoneleaf.us>
 <mailman.186.1482285005.2322.python-list@python.org>
Message-ID: <o3j0u8$df9$1@news.icm.edu.pl>

W dniu 21.12.2016 o 02:51, Ethan Furman pisze:
> On 12/20/2016 03:39 PM, Ben Finney wrote:
>> "Mr. Wrobel" writes:
>
>>> Quick question, can anybody tell me when to use __init__ instead of
>>> __new__ in meta programming?
>>
>> Use ?__new__? to do the work of *creating* one instance from nothing;
>> allocating the storage, determining the type, etc. ? anything that will
>> be *the same* for every instance. ?__new__? is the constructor.
>>
>> Use ?__init__? to do the work of *configuring* one instance, after it
>> already exists. Any attributes that are special to an instance should be
>> manipulated in the ?__init__? method. ?__init__? is the initialiser.
>
> That sounds like general object creation/class advice, which as a general
> guideline is okay, but don't feel like it's the most important thing.
>
> I only use `__new__` when the object being created is (or is based on)
> an immutable type; otherwise I use `__init__`.  Likewise, if I'm using
> `__new__` then I do all my configuration in `__new__` unless I have a
> really good reason not to (such as making it easier for subclasses to
> modify/skip `__init__`).
>
> As far as metaclasses go... the only time I recall writing an `__init__`
> for a metaclass was to strip off the extra arguments so `type.__init__`
> wouldn't fail.
>
> --
> ~Ethan~
Hi,thanx for answers, let's imagine that we want to add one class 
attribute for newly created classess with using __init__ in metaclass, 
here's an example:

#!/usr/bin/env python

class MetaClass(type):
     # __init__ manipulation:

     def __init__(cls, name, bases, dct):
         dct['added_in_init'] = 'test'
         super(MetaClass, cls).__init__(name, bases, dct)

class BaseClass(object):
     __metaclass__ = MetaClass

class NewBaseClass(BaseClass):
     pass

print("Lets print attributes added in __init__ in base classes:")

print(BaseClass.added_in_init)

print(NewBaseClass.added_in_init)

after running it: AttributeError: type object 'BaseClass' has no 
attribute 'added_in_init'

Adding the same in __new__ works. Can anyone explain me please what's wrong?

Cheers,
M





From rosuav at gmail.com  Fri Dec 23 06:26:17 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 23 Dec 2016 22:26:17 +1100
Subject: Another security question
In-Reply-To: <o3itmj$t42$1@blaine.gmane.org>
References: <o3itmj$t42$1@blaine.gmane.org>
Message-ID: <CAPTjJmpPPGM+_ut_AMTNb7vgo0vRgPtu6iAgYjqWVpXG5ypd_A@mail.gmail.com>

On Fri, Dec 23, 2016 at 9:19 PM, Frank Millman <frank at chagford.com> wrote:
> At present I just store a SHA-1 hash of the password for each user. Here are
> my thoughts on improving this.
>
> 1. Generate a 'salt' for each password. There seem to be two ways in the
> standard library to do this -
>    import os
>    salt = os.urandom(16)
>
>    import secrets
>    salt = secrets.token_bytes(16)
>
>    My guess is that it will not make much difference which I use.

The main difference is that the 'secrets' module is new in Python 3.6.
If you use anything older - and there are a lot of 3.5s and 3.4s out
there - you can't use it (unless there's a PyPI backport or
something). So if you need compatibility with older Pythons, use
os.urandom; if you're okay with 3.6+, use secrets.

> 2. Store the salt in the database along with the user-id and hashed password
> for each user.

Yep. I generally work with a single database field containing the salt
and the hash as a combined "encrypted password", as it's convenient to
work that way. It's also often worth storing some kind of signature so
you know what scheme you used; in the future, you will eventually
decide that your passwords aren't really secure enough for long-term,
and you'll want to progressively change over. You can't do that by
decrypting and re-encrypting the passwords (since you can't decrypt
them), so you have to introduce a new scheme while still supporting
the old one. Technically you could detect the scheme by the encrypted
length, but it's easier and safer to have a signature.

> 3. Generate the password from the string supplied by the user as follows -
>    from hashlib import blake2b
>    password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for hashing
> passwords. See BLAKE2 FAQ for more information."
>
> I propose to ignore this warning. I feel that, for my purposes, the above
> procedure is adequate.
>
> Does all this sound reasonable?

Check out some prior art. When I build a web app using Flask, I
generally use Werkzeug's password management features:

http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.generate_password_hash
http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.check_password_hash

As well as doing everything I said above about salting and hashing and
having signatures, it pushes the responsibility onto someone else. You
just give it a password and get back an ASCII string that you stash in
the database. If there's a security flaw, Werkzeug can push a new
version that fixes it - it's not your problem.

At very least, be aware of what these kinds of libraries are doing.
I'm not saying you should blindly trust them or automatically reach
for a dependency, but they're worth looking at.

ChrisA


From guohaochuan at gmail.com  Fri Dec 23 07:39:17 2016
From: guohaochuan at gmail.com (Haochuan Guo)
Date: Fri, 23 Dec 2016 12:39:17 +0000
Subject: Garbage collection problem with generators
Message-ID: <CAF=uXRWep5YTjUGxCB2fFO-V4kL6PC5ymadP_em68mYXybZaVg@mail.gmail.com>

Hi, everyone

I'm building a http long polling client for our company's discovery service
and something weird happened in the following code:

```python
while True:
    try:
        r = requests.get("url", stream=True, timeout=3)
        for data in r.iter_lines():
            processing_data...
    except TimeoutException:
        time.sleep(10)
```

When I deliberately times out the request and then check the connections
with `lsof -p process`, I discover that there are *two active
connections*(ESTABLISH)
instead of one. After digging around, it turns out it might not be the
problem with `requests` at all, but gc related to generators.

So I write this script to demonstrate the problem:

https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef

Function `A.a` will return a generator which will raise an exception. And
in function `b`, I'm building new a new instance of `A` and iterate over
the exception-raising generator. In the exception handler, I'll close the
generator, delete it, delete the `A` instance, call `gc.collect()` and do
the whole process all over again.

There's another greenlet checking the `A` instances by using
`gc.get_objects()`. It turns out there are always two `A` instances.

This is reproducible with python2.7, but not in python3.5. I've also tried
with `thread` instead of `gevent`, it still happens. I'm guessing it's
related to garbage collection of generators.

Did I bump into a python2 bug? Or am I simply wrong about the way to close
generators...?

Thanks


From robin at reportlab.com  Fri Dec 23 07:43:05 2016
From: robin at reportlab.com (Robin Becker)
Date: Fri, 23 Dec 2016 12:43:05 +0000
Subject: [RELEASE] Python 3.6.0 is released!
In-Reply-To: <1C5335CD-4028-4F1C-A5EC-6A4DA615552E@python.org>
References: <1C5335CD-4028-4F1C-A5EC-6A4DA615552E@python.org>
Message-ID: <6022b26b-6a34-5326-58b9-bade505ce65f@chamonix.reportlab.co.uk>

On 23/12/2016 10:34, Ned Deily wrote:
> On behalf of the Python development community and the Python 3.6 release
> team, I am pleased to announce the availability of Python 3.6.0.  Python
> 3.6.0 is the newest major release of the Python language, and it contains
> many new features and optimizations.  See the "What?s New In Python 3.6"
> document for more information:
>
>     https://docs.python.org/3.6/whatsnew/3.6.html
>
> You can download Python 3.6.0 here:
>
>     https://www.python.org/downloads/release/python-360/
>
> Also, most third-party distributors of Python should be making 3.6.0
> packages available soon.
>
> Maintenance releases for the 3.6 series will follow at regular intervals
> starting in the first quarter of 2017.
>
> We hope you enjoy Python 3.6.0!
>.......

Thanks for this release; makes my Christmas just a bit more stressed :( or 
perhaps happier :)

Anyhow I am trying to figure out this error when building windows versions


C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: 
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
   warn=(impl == 'cp')):
C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:81: RuntimeWarning: 
Config variable 'WITH_PYMALLOC' is unset,
Python ABI tag may be incorrect
   warn=(impl == 'cp')):

I guess this must mean I need to set something somewhere, but what?
-- 
Robin Becker



From ben.usenet at bsb.me.uk  Fri Dec 23 08:33:53 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Fri, 23 Dec 2016 13:33:53 +0000
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
Message-ID: <87fulebwhq.fsf@bsb.me.uk>

"Frank Millman" <frank at chagford.com> writes:
<snip>
> ... Here are my thoughts on improving this.
>
> 1. Generate a 'salt' for each password. There seem to be two ways in
> the standard library to do this -
>    import os
>    salt = os.urandom(16)
>
>    import secrets
>    salt = secrets.token_bytes(16)
>
>    My guess is that it will not make much difference which I use.
>
> 2. Store the salt in the database along with the user-id and hashed
> password for each user.
>
> 3. Generate the password from the string supplied by the user as follows -
>    from hashlib import blake2b
>    password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other
> general-purpose cryptographic hash function, such as SHA-256, is not
> suitable for hashing passwords. See BLAKE2 FAQ for more information."

As stated, this is confusing as BLAKE2's site lists several password
hashing schemes that use it!  The point is that you should not use
*only* a simple salted hash because it's too efficient and therefore
open to brute-force attacks.  The hashing schemes that use BLAKE2 are
deliberately designed to be costly.

> I propose to ignore this warning. I feel that, for my purposes, the
> above procedure is adequate.
>
> Does all this sound reasonable?

That depends on the purposes, of course, so it's hard to offer advice.

-- 
Ben.


From frank at chagford.com  Fri Dec 23 08:52:41 2016
From: frank at chagford.com (Frank Millman)
Date: Fri, 23 Dec 2016 15:52:41 +0200
Subject: Another security question
In-Reply-To: <CAPTjJmpPPGM+_ut_AMTNb7vgo0vRgPtu6iAgYjqWVpXG5ypd_A@mail.gmail.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <CAPTjJmpPPGM+_ut_AMTNb7vgo0vRgPtu6iAgYjqWVpXG5ypd_A@mail.gmail.com>
Message-ID: <o3ja78$g97$1@blaine.gmane.org>

"Chris Angelico"  wrote in message 
news:CAPTjJmpPPGM+_ut_AMTNb7vgo0vRgPtu6iAgYjqWVpXG5ypd_A at mail.gmail.com...
>
> On Fri, Dec 23, 2016 at 9:19 PM, Frank Millman <frank at chagford.com> wrote:
>
> > 3. Generate the password from the string supplied by the user as 
> > follows -
> >    from hashlib import blake2b
> >    password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
> >
> > The hashlib docs have the following warning -
> >
> > "Salted hashing (or just hashing) with BLAKE2 or any other 
> > general-purpose
> > cryptographic hash function, such as SHA-256, is not suitable for 
> > hashing
> > passwords. See BLAKE2 FAQ for more information."
> >
> > I propose to ignore this warning. I feel that, for my purposes, the 
> > above
> > procedure is adequate.
> >
> > Does all this sound reasonable?
>
> Check out some prior art. When I build a web app using Flask, I
> generally use Werkzeug's password management features:
>
> http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.generate_password_hash
> http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.check_password_hash
>
> As well as doing everything I said above about salting and hashing and
> having signatures, it pushes the responsibility onto someone else. You
> just give it a password and get back an ASCII string that you stash in
> the database. If there's a security flaw, Werkzeug can push a new
> version that fixes it - it's not your problem.
>
> At very least, be aware of what these kinds of libraries are doing.
> I'm not saying you should blindly trust them or automatically reach
> for a dependency, but they're worth looking at.
>

All excellent advice - thanks very much.

It seems that Werkzeug (which looks great, by the way) uses something called 
pbkdf2.

The new kid on the block seems to be Argon2. A python implementation called 
argon2_cffi has been released by Hynek Schlawack, who has written this 
article -
    https://hynek.me/articles/storing-passwords/

This is his preamble - "if you?re hashing your passwords with 
bcrypt/scrypt/PBKDF2 today, there?s nothing to worry about in the immediate 
future. This article is for you if you?re choosing a password hash today and 
want a future-proof solution."

I eventually got argon2_cffi installed, and it works very nicely, so I will 
run with that for now.

Thanks again

Frank





From ian.g.kelly at gmail.com  Fri Dec 23 09:14:43 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 23 Dec 2016 08:14:43 -0600
Subject: Metaclasses - magic functions
In-Reply-To: <o3j0u8$df9$1@news.icm.edu.pl>
References: <o3c6d8$sru$1@news.icm.edu.pl> <85a8bqxj9m.fsf@benfinney.id.au>
 <5859E013.8010108@stoneleaf.us>
 <mailman.186.1482285005.2322.python-list@python.org>
 <o3j0u8$df9$1@news.icm.edu.pl>
Message-ID: <CALwzid=6fjZ2EgfpVrQ6FVMazRTw3ny2Vj3pAWfuNPfRVpSCvQ@mail.gmail.com>

On Fri, Dec 23, 2016 at 5:14 AM, Mr. Wrobel <mr at e-wrobel.pl> wrote:
> Hi,thanx for answers, let's imagine that we want to add one class attribute
> for newly created classess with using __init__ in metaclass, here's an
> example:
>
> #!/usr/bin/env python
>
> class MetaClass(type):
>     # __init__ manipulation:
>
>     def __init__(cls, name, bases, dct):
>         dct['added_in_init'] = 'test'
>         super(MetaClass, cls).__init__(name, bases, dct)
>
> class BaseClass(object):
>     __metaclass__ = MetaClass
>
> class NewBaseClass(BaseClass):
>     pass
>
> print("Lets print attributes added in __init__ in base classes:")
>
> print(BaseClass.added_in_init)
>
> print(NewBaseClass.added_in_init)
>
> after running it: AttributeError: type object 'BaseClass' has no attribute
> 'added_in_init'
>
> Adding the same in __new__ works. Can anyone explain me please what's wrong?

When __init__ is called, the class has already been constructed by
__new__, and the 'dct' argument has already been copied into the class
dict. The base __init__ method does nothing, so adding the item to dct
and calling up doesn't accomplish anything.

Instead, the 'cls' argument that gets passed into __init__ is the
newly constructed class, so just use that to set the attributes:

    cls.added_in_init = 'test'


From larry.martell at gmail.com  Fri Dec 23 11:30:25 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Fri, 23 Dec 2016 11:30:25 -0500
Subject: dateutil timezone question
Message-ID: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>

I have a datetime that looks like this: '2016-11-11T18:10:09-05:00'
and when I pass it to dateutil.parser.parse I get back this:

datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000))

And I have other datetimes like this: '2016-04-27T00:00:00', which
went passed to dateutil.parser.parse of course does not return a
datetime with the tzinfo.

I need to compare these datetimes, and if I do that I get the dreaded
"can't compare offset-naive and offset-aware datetimes" error.

Is there a way I can get it back without the tzinfo, but instead with
the offset applied to the date, so I can compare these 2?

In other words I want it to return

datetime.datetime(2016, 11, 11, 13, 10, 9)


From steve+python at pearwood.info  Fri Dec 23 11:58:59 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 24 Dec 2016 03:58:59 +1100
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o3j0jt$3pc$1@blaine.gmane.org>
 <mailman.223.1482491346.2322.python-list@python.org>
Message-ID: <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>

On Fri, 23 Dec 2016 10:08 pm, Frank Millman wrote:

> "Steve D'Aprano"  wrote in message
> news:585d009f$0$1599$c3e8da3$5496439d at news.astraweb.com...
>>
>> On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:
>>
>> >
>> > 3. Generate the password from the string supplied by the user as
>> > follows -
>> >     from hashlib import blake2b
>> >     password = blake2b('my_password'.encode('utf-8'),
>> > salt=salt).digest()
>> >
>> > The hashlib docs have the following warning -
>> >
>> > "Salted hashing (or just hashing) with BLAKE2 or any other
>> > general-purpose
>> > cryptographic hash function, such as SHA-256, is not suitable for
>> > hashing
>> > passwords. See BLAKE2 FAQ for more information."
>>
>> Why are using Blake2 when the docs explicitly say not to use them in this
>> way? Have you read the FAQ to see what it says?
>>
> 
> Why am I using Blake2? Well, before today I had not heard of it. However,
> in the past, when I needed to create a hashed password, I used the
> built-in hashlib module. Today, when I look at the docs for hashlib in
> Python 3.6, this is the new sub-heading -
>     "15.2. hashlib ? BLAKE2 hash functions"
> 
> So it appears that this is the new preferred way of doing it.

What makes a good hash function for passwords is not the same as a good
general purpose hash function, cryptographic or not.

You can read more about this:

http://security.blogoverflow.com/2013/09/about-secure-password-hashing/
https://crackstation.net/hashing-security.htm
http://www.darkreading.com/safely-storing-user-passwords-hashing-vs-encrypting/a/d-id/1269374

but the TL;DR is that any of the general-purpose hash functions -- md5,
sha-1, sha-2 (sha-256 or sha-512), sha-3 or BLAKE2 -- are poor choices
because they're *too fast*, or have other vulnerabilities, or both.


> This is what the Blake2 FAQ says -
> 
> "You shouldn't use *any* general-purpose hash function for user passwords,
> not BLAKE2, and not MD5, SHA-1, SHA-256, or SHA-3. Instead you should use
> a password hashing function such as the PHC winner Argon2 with appropriate
> time and memory cost parameters, to mitigate the risk of bruteforce
> attacks?Argon2's core uses a variant of BLAKE2's permutation"
> 
> I see that there is a Python implementation of Argon2 in PyPi, but I don't
> particularly want to add another dependency to my app. My gut-feel says
> that this is overkill for my requirement. However, I am not sure. That is
> partly why I started this thread, to get some counter-arguments.

I have no opinion about Argon2, but I too would be reluctant to use an
external dependency of unknown quality.

The tried and tested password hashing functions are PBKDF2, bcrypt and
scrypt, with bcrypt generally considered the "boring, reliable" solution.
But there's no Python standard library implementation, which is sad.

crypt is also said to be sufficiently strong, but only some versions, and I
believe it is Unix/Linux only.

https://docs.python.org/3.5/library/crypt.html#module-crypt


There is a stdlib PBKDF2. If you want to avoid third-party dependencies, use
that.

https://docs.python.org/3.4/library/hashlib.html#hashlib.pbkdf2_hmac


By the way, thanks for raising this interesting question! This is exactly
the sort of thing that the secrets module is supposed to make a "no
brainer", so I expect that it will get a password hash function.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Fri Dec 23 14:18:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 06:18:46 +1100
Subject: dateutil timezone question
In-Reply-To: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
References: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
Message-ID: <CAPTjJmquKGexpTKLvJ+=WujZsE=UM4qLv_aKLkuA0ksiea_vOw@mail.gmail.com>

On Sat, Dec 24, 2016 at 3:30 AM, Larry Martell <larry.martell at gmail.com> wrote:
> I have a datetime that looks like this: '2016-11-11T18:10:09-05:00'
> and when I pass it to dateutil.parser.parse I get back this:
>
> datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000))
>
> And I have other datetimes like this: '2016-04-27T00:00:00', which
> went passed to dateutil.parser.parse of course does not return a
> datetime with the tzinfo.
>
> I need to compare these datetimes, and if I do that I get the dreaded
> "can't compare offset-naive and offset-aware datetimes" error.

Some of your timestamps have timezones and others don't. That's a
fundamental problem. Are you absolutely certain that the ones without
them are in UTC? If so, the easiest fix would be to slap a "Z" on them
before you parse, which would give you exclusively aware datetimes.

ChrisA


From mr at e-wrobel.pl  Fri Dec 23 14:20:41 2016
From: mr at e-wrobel.pl (Mr. Wrobel)
Date: Fri, 23 Dec 2016 20:20:41 +0100
Subject: Metaclasses - magic functions
In-Reply-To: <mailman.231.1482502526.2322.python-list@python.org>
References: <o3c6d8$sru$1@news.icm.edu.pl> <85a8bqxj9m.fsf@benfinney.id.au>
 <5859E013.8010108@stoneleaf.us>
 <mailman.186.1482285005.2322.python-list@python.org>
 <o3j0u8$df9$1@news.icm.edu.pl>
 <CALwzid=6fjZ2EgfpVrQ6FVMazRTw3ny2Vj3pAWfuNPfRVpSCvQ@mail.gmail.com>
 <mailman.231.1482502526.2322.python-list@python.org>
Message-ID: <o3jte9$4j4$1@news.icm.edu.pl>

W dniu 23.12.2016 o 15:14, Ian Kelly pisze:
(...)
>
>     cls.added_in_init = 'test'
>

Man, you are awsome genius! Finally somebody was able to explain me what 
is the power of __new__ and difference between __init__ !!!

So what I wanted to achieve was adding some new attributes to the class 
instances (objects) like this:

########
#!/usr/bin/env python

class MetaClass(type):
     # __init__ manipulation:

     def __init__(cls, name, bases, dct):
         cls.added_in_init = 'test'
         super(MetaClass, cls).__init__(name, bases, dct)

class BaseClass(object):
     __metaclass__ = MetaClass

class NewBaseClass(BaseClass):
     def __init__(self):
         self.inst_attr = self.added_in_init


print("Lets print attributes added in __init__ in base classes:")


b = NewBaseClass()
print(b.added_in_init)

print(b.inst_attr)
########

And finally it works!

Man I owe you a galaxy of thanks, because I was so frustrated! The 
examples taken from internet never explained me it in so clearly like 
you did!

Really, really thank you!!!


From rosuav at gmail.com  Fri Dec 23 14:24:16 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 06:24:16 +1100
Subject: Another security question
In-Reply-To: <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o3j0jt$3pc$1@blaine.gmane.org>
 <mailman.223.1482491346.2322.python-list@python.org>
 <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmovU1Vcxdu-ZXnQDbVNGDEyx+oD2HXVHr6QSZ7sgBWp4g@mail.gmail.com>

On Sat, Dec 24, 2016 at 3:58 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> By the way, thanks for raising this interesting question! This is exactly
> the sort of thing that the secrets module is supposed to make a "no
> brainer", so I expect that it will get a password hash function.

+1. Please can we see something like Werkzeug's functions added?
They're the no-brainer option for anything that already has Werkzeug
as a dependency (eg Flask apps), and if that were in the stdlib,
they'd be the no-brainer option for 3.7+ programs.

Should I take this to -ideas?

ChrisA


From skip.montanaro at gmail.com  Fri Dec 23 14:24:19 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 23 Dec 2016 13:24:19 -0600
Subject: US/Eastern offset
In-Reply-To: <CAPTjJmqtJq8yfQEZ=jFWs3ApA+485dexFEcRBXado=2oa6-aGg@mail.gmail.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <o3hokk$cl8$1@blaine.gmane.org>
 <mailman.216.1482450413.2322.python-list@python.org>
 <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>
 <CAPTjJmqtJq8yfQEZ=jFWs3ApA+485dexFEcRBXado=2oa6-aGg@mail.gmail.com>
Message-ID: <CANc-5UybmFmvbKiksuwxK4jFrA5UPgFO1fziLBJ23t2jPAnhxg@mail.gmail.com>

Okay, problem solved. My thanks to Chris Barker over on the Anaconda group
for help. (I originally thought there might be something amiss with the
pytz package in Anaconda, as our older non-Anaconda Python seemed not to
have the problem.) It turns out to be a problem I solved several years ago
at my previous employer (but forgot I had solved, and for perhaps obvious
reasons, I wasn't able to take my solution with me - another advantage for
open source...). It all boils down to the difference between these two
operations:

now.replace(tzinfo=eastern_tz)
eastern_tz.astimezone(now)

where now is datetime.datetime.now() and eastern_tz is
pytz.timezone("America/Eastern").

So, my problem has (again) been solved. A colleague here has suggested arrow
<https://pypi.python.org/pypi/arrow/0.10.0> as an alternative to
datetime+pytz, which I will look into during the slow period next week.

Skip


From skip.montanaro at gmail.com  Fri Dec 23 14:27:49 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 23 Dec 2016 13:27:49 -0600
Subject: dateutil timezone question
In-Reply-To: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
References: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
Message-ID: <CANc-5Uy5d+nkJ_9ynTYCYo=8hbrjJq77PTKTtaDANb0=pSNNFQ@mail.gmail.com>

> I need to compare these datetimes, and if I do that I get the dreaded
> "can't compare offset-naive and offset-aware datetimes" error.

If you're sure the naive datetimes are UTC, this should work:

import pytz

dt = pytz.utc.localize(dateutil.parser.parse('2016-04-27T00:00:00'))

You can then compare it with other tz-containing datetime objects.

Skip


From larry.martell at gmail.com  Fri Dec 23 14:32:54 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Fri, 23 Dec 2016 14:32:54 -0500
Subject: dateutil timezone question
In-Reply-To: <CAPTjJmquKGexpTKLvJ+=WujZsE=UM4qLv_aKLkuA0ksiea_vOw@mail.gmail.com>
References: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
 <CAPTjJmquKGexpTKLvJ+=WujZsE=UM4qLv_aKLkuA0ksiea_vOw@mail.gmail.com>
Message-ID: <CACwCsY5JLYwLohDegdmCh9VrD5saKZ5Htr2Q4fZwFq+sy_9hpg@mail.gmail.com>

On Fri, Dec 23, 2016 at 2:18 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Dec 24, 2016 at 3:30 AM, Larry Martell <larry.martell at gmail.com> wrote:
>> I have a datetime that looks like this: '2016-11-11T18:10:09-05:00'
>> and when I pass it to dateutil.parser.parse I get back this:
>>
>> datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000))
>>
>> And I have other datetimes like this: '2016-04-27T00:00:00', which
>> went passed to dateutil.parser.parse of course does not return a
>> datetime with the tzinfo.
>>
>> I need to compare these datetimes, and if I do that I get the dreaded
>> "can't compare offset-naive and offset-aware datetimes" error.
>
> Some of your timestamps have timezones and others don't. That's a
> fundamental problem. Are you absolutely certain that the ones without
> them are in UTC? If so, the easiest fix would be to slap a "Z" on them
> before you parse, which would give you exclusively aware datetimes.

What I ended up doing, which I think is somewhat kludgy is this:

def add_tz(datestr):
    if datestr is None or len(datestr) < 6:
        return datestr
    if datestr[-6] == ':':
        return datestr+'-00:00'
    else:
        return datestr

Then I pass all my dates through that before calling dateutil.parser.parse


From larry.martell at gmail.com  Fri Dec 23 14:43:29 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Fri, 23 Dec 2016 14:43:29 -0500
Subject: dateutil timezone question
In-Reply-To: <CANc-5Uy5d+nkJ_9ynTYCYo=8hbrjJq77PTKTtaDANb0=pSNNFQ@mail.gmail.com>
References: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
 <CANc-5Uy5d+nkJ_9ynTYCYo=8hbrjJq77PTKTtaDANb0=pSNNFQ@mail.gmail.com>
Message-ID: <CACwCsY6Zo=yaSG1mjbhvi3w4wszFd3G5KUJL6=WgoE6Rscxp+A@mail.gmail.com>

On Fri, Dec 23, 2016 at 2:27 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
>> I need to compare these datetimes, and if I do that I get the dreaded
>> "can't compare offset-naive and offset-aware datetimes" error.
>
> If you're sure the naive datetimes are UTC, this should work:
>
> import pytz
>
> dt = pytz.utc.localize(dateutil.parser.parse('2016-04-27T00:00:00'))
>
> You can then compare it with other tz-containing datetime objects.

I did mess around with pytz a bit but I was getting a lot of
exceptions - something related to the TZ already being set or
something like that. I don't recall exactly, and I can't scroll back
far enough to find it.


From jladasky at itu.edu  Fri Dec 23 14:51:11 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Fri, 23 Dec 2016 11:51:11 -0800 (PST)
Subject: US/Eastern offset
In-Reply-To: <mailman.217.1482458251.2322.python-list@python.org>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <o3hokk$cl8$1@blaine.gmane.org>
 <mailman.216.1482450413.2322.python-list@python.org>
 <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>
 <CAPTjJmqtJq8yfQEZ=jFWs3ApA+485dexFEcRBXado=2oa6-aGg@mail.gmail.com>
 <mailman.217.1482458251.2322.python-list@python.org>
Message-ID: <5a46ccf2-f8cd-40f9-a865-b488fab5bf8f@googlegroups.com>

On Thursday, December 22, 2016 at 5:57:42 PM UTC-8, Chris Angelico wrote:
> On Fri, Dec 23, 2016 at 12:54 PM,  <j... at itu.edu> wrote:
> > Wouldn't most users prefer that modern time zones be the default information returned by pytz, instead of 150 year-old historical time zones?
> 
> They're the *same* time zones.
> 
> ChrisA

Apparently, they're not quite the same.  The four-minute discrepancy between New York local (mean solar?) time and the modern time zone is what got Skip Montanaro asking questions.


From skip.montanaro at gmail.com  Fri Dec 23 15:09:21 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 23 Dec 2016 12:09:21 -0800 (PST)
Subject: dateutil timezone question
In-Reply-To: <mailman.238.1482522259.2322.python-list@python.org>
References: <CACwCsY4g0kcxvuN9nNLC0eopoNzQKDVreYWg9E19HqjsduhLig@mail.gmail.com>
 <CANc-5Uy5d+nkJ_9ynTYCYo=8hbrjJq77PTKTtaDANb0=pSNNFQ@mail.gmail.com>
 <CACwCsY6Zo=yaSG1mjbhvi3w4wszFd3G5KUJL6=WgoE6Rscxp+A@mail.gmail.com>
 <mailman.238.1482522259.2322.python-list@python.org>
Message-ID: <aa546300-a87b-4d0d-8ad2-3d1933560049@googlegroups.com>

> I did mess around with pytz a bit but I was getting a lot of
> exceptions - something related to the TZ already being set or
> something like that. I don't recall exactly, and I can't scroll back
> far enough to find it.

Yes, if the tzinfo attribute has already been set, you will get errors from localize(). Once a datetime object has a valid timezone, you can compare it with other tz-aware datetime objects. They need not be in the same timezone.

For visual purposes (for instance, when writing to logfiles), it can be handy to display all timestamps in the same timezone. For that, you can use the normalize() method of timezone objects. Here's a simple example:

>>> import datetime, pytz

Here's a datetime in UTC, but still naive (no tzinfo attribute).

>>> utcnow = datetime.datetime.utcnow()
>>> print utcnow.tzinfo
None

Localize to UTC et voila!

>>> utcnow = pytz.utc.localize(utcnow)
>>> print utcnow.tzinfo
UTC
>>> chicago = pytz.timezone("America/Chicago")
>>> print utcnow
2016-12-23 20:04:51.295127+00:00
>>> print chicago.normalize(utcnow)
2016-12-23 14:04:51.295127-06:00

Different timezones, but the same time.

>>> print utcnow - chicago.normalize(utcnow)
0:00:00

Skip


From valkrem at yahoo.com  Fri Dec 23 15:39:59 2016
From: valkrem at yahoo.com (Val Krem)
Date: Fri, 23 Dec 2016 20:39:59 +0000 (UTC)
Subject: data frame
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
Message-ID: <235400868.1750803.1482525599916@mail.yahoo.com>

Hi all,

#!/usr/bin/env python
import sys
import csv
import numpy as np
import pandas as  pd

a= pd.read_csv("s1.csv")
print(a)

     size   w1   h1
0  512  214   26
1  123  250   34
2  234  124   25
3  334  213   43
4  a45  223   32
5  a12  214   26

I wanted to create a new column by adding the two column values 
as follows

a['test'] = a['w1'] + a['h1']

Traceback (most recent call last):
File "/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py", line 2104, in get_loc
return self._engine.get_loc(key)
File "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc (pandas/index.c:4152)
File "pandas/index.pyx", line 161, in pandas.index.IndexEngine.get_loc (pandas/index.c:4016)
File "pandas/src/hashtable_class_helper.pxi", line 732, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
KeyError: 'w1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "tt.py", line 16, in <module>
a['test']=a['w1'] + a['h1']

File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
KeyError: 'w1'

Can someone help me what the problem is?

Thank you in advance


From rosuav at gmail.com  Fri Dec 23 15:50:01 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 07:50:01 +1100
Subject: US/Eastern offset
In-Reply-To: <5a46ccf2-f8cd-40f9-a865-b488fab5bf8f@googlegroups.com>
References: <CANc-5UzAQfy-dsva6ovRUGDD1zGo39QFXxsnx8h8aK1rQTdDTQ@mail.gmail.com>
 <o3hokk$cl8$1@blaine.gmane.org>
 <mailman.216.1482450413.2322.python-list@python.org>
 <fa1bcec5-bccd-44b9-a059-1ed06819548b@googlegroups.com>
 <CAPTjJmqtJq8yfQEZ=jFWs3ApA+485dexFEcRBXado=2oa6-aGg@mail.gmail.com>
 <mailman.217.1482458251.2322.python-list@python.org>
 <5a46ccf2-f8cd-40f9-a865-b488fab5bf8f@googlegroups.com>
Message-ID: <CAPTjJmoay37ar=49TD8gBwKvbbKoTZV5nK+N_kcQsnurNF2FqQ@mail.gmail.com>

On Sat, Dec 24, 2016 at 6:51 AM,  <jladasky at itu.edu> wrote:
> On Thursday, December 22, 2016 at 5:57:42 PM UTC-8, Chris Angelico wrote:
>> On Fri, Dec 23, 2016 at 12:54 PM,  <j... at itu.edu> wrote:
>> > Wouldn't most users prefer that modern time zones be the default information returned by pytz, instead of 150 year-old historical time zones?
>>
>> They're the *same* time zones.
>>
>> ChrisA
>
> Apparently, they're not quite the same.  The four-minute discrepancy between New York local (mean solar?) time and the modern time zone is what got Skip Montanaro asking questions.

Is "US/Eastern" the same thing as "America/New_York"? According to my
system, they are symlinks to the same content. So they are actually
the same time zone. It's a time zone that has different UTC offsets at
different points in time, both historic (the change from local to
standard time) and modern (a twice-yearly shift in clocks), but it's
the same timezone. If you look at any other form that means "New_York"
(eg "EST"), and then set the date way back, you should see the same
phenomenon.

ChrisA


From rosuav at gmail.com  Fri Dec 23 15:53:31 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 07:53:31 +1100
Subject: data frame
In-Reply-To: <235400868.1750803.1482525599916@mail.yahoo.com>
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
 <235400868.1750803.1482525599916@mail.yahoo.com>
Message-ID: <CAPTjJmp=g66AX9HWXLHwiobW4F5UAeDf62TcX6UJnqT5FpTnpw@mail.gmail.com>

On Sat, Dec 24, 2016 at 7:39 AM, Val Krem via Python-list
<python-list at python.org> wrote:
> a= pd.read_csv("s1.csv")
> File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> Can someone help me what the problem is?

Can you post the first line of the CSV file? Maybe it isn't 'w1' exactly.

ChrisA


From __peter__ at web.de  Fri Dec 23 16:08:28 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 23 Dec 2016 22:08:28 +0100
Subject: data frame
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
 <235400868.1750803.1482525599916@mail.yahoo.com>
Message-ID: <o3k3o7$onp$1@blaine.gmane.org>

Val Krem via Python-list wrote:

> Hi all,
> 
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as  pd
> 
> a= pd.read_csv("s1.csv")
> print(a)
> 
>      size   w1   h1
> 0  512  214   26
> 1  123  250   34
> 2  234  124   25
> 3  334  213   43
> 4  a45  223   32
> 5  a12  214   26
> 
> I wanted to create a new column by adding the two column values
> as follows
> 
> a['test'] = a['w1'] + a['h1']
> 
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
> File "tt.py", line 16, in <module>
> a['test']=a['w1'] + a['h1']
> 
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
> 
> Can someone help me what the problem is?
> 
> Thank you in advance

Have a look at a.keys(). I suspect that the column name has extra space:

>>> pd.read_csv("s1.csv").keys()
Index([u'size', u' w1', u' h1'], dtype='object')

I that's what you see you can fix it by reading the csv with 
skipinitialspace=True:

>>> pd.read_csv("s1.csv", skipinitialspace=True).keys()
Index([u'size', u'w1', u'h1'], dtype='object')




From valkrem at yahoo.com  Fri Dec 23 16:55:44 2016
From: valkrem at yahoo.com (Val Krem)
Date: Fri, 23 Dec 2016 21:55:44 +0000 (UTC)
Subject: data frame
In-Reply-To: <o3k3o7$onp$1@blaine.gmane.org>
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
 <235400868.1750803.1482525599916@mail.yahoo.com>
 <o3k3o7$onp$1@blaine.gmane.org>
Message-ID: <1242451632.1794626.1482530144997@mail.yahoo.com>

Here is the first few lines of the data


s1.csv 
size,w1,h1
512,214,26
123,250,34
234,124,25
334,213,43

and the script

    a=pd.read_csv("s1.csv", skipinitialspace=True).keys()
    print(a)
i see the following

Index(['size', 'w1', 'h1'], dtype='object')



when I wanted to add the two columns; then I get the following message.

a=pd.read_csv("s1.csv", skipinitialspace=True).keys()
a['test']=a['w1'] + a['h1']
print(a)




data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py:1393: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
return getitem(key)
Traceback (most recent call last):
File "tt.py", line 12, in <module>
a['test']=a['w1'] + a['h1']
File "/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py", line 1393, in __getitem__
return getitem(key)
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices



On Friday, December 23, 2016 3:09 PM, Peter Otten <__peter__ at web.de> wrote:
Val Krem via Python-list wrote:

> Hi all,
>
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as  pd
>
> a= pd.read_csv("s1.csv")
> print(a)
>
>      size  w1  h1
> 0  512  214  26
> 1  123  250  34
> 2  234  124  25
> 3  334  213  43
> 4  a45  223  32
> 5  a12  214  26
>
> I wanted to create a new column by adding the two column values
> as follows
>
> a['test'] = a['w1'] + a['h1']
>
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "tt.py", line 16, in <module>
> a['test']=a['w1'] + a['h1']
>
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> Can someone help me what the problem is?
>
> Thank you in advance

Have a look at a.keys(). I suspect that the column name has extra space:

>>> pd.read_csv("s1.csv").keys()
Index([u'size', u' w1', u' h1'], dtype='object')

I that's what you see you can fix it by reading the csv with
skipinitialspace=True:

>>> pd.read_csv("s1.csv", skipinitialspace=True).keys()
Index([u'size', u'w1', u'h1'], dtype='object')


-- 
https://mail.python.org/mailman/listinfo/python-list


/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py:1393: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
return getitem(key)
Traceback (most recent call last):
File "tt.py", line 12, in <module>
a['test']=a['w1'] + a['h1']
File "/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py", line 1393, in __getitem__
return getitem(key)
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices







On Friday, December 23, 2016 3:09 PM, Peter Otten <__peter__ at web.de> wrote:
Val Krem via Python-list wrote:

> Hi all,
> 
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as  pd
> 
> a= pd.read_csv("s1.csv")
> print(a)
> 
>      size   w1   h1
> 0  512  214   26
> 1  123  250   34
> 2  234  124   25
> 3  334  213   43
> 4  a45  223   32
> 5  a12  214   26
> 
> I wanted to create a new column by adding the two column values
> as follows
> 
> a['test'] = a['w1'] + a['h1']
> 
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
> File "tt.py", line 16, in <module>
> a['test']=a['w1'] + a['h1']
> 
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
> 
> Can someone help me what the problem is?
> 
> Thank you in advance

Have a look at a.keys(). I suspect that the column name has extra space:

>>> pd.read_csv("s1.csv").keys()
Index([u'size', u' w1', u' h1'], dtype='object')

I that's what you see you can fix it by reading the csv with 
skipinitialspace=True:

>>> pd.read_csv("s1.csv", skipinitialspace=True).keys()
Index([u'size', u'w1', u'h1'], dtype='object')


-- 
https://mail.python.org/mailman/listinfo/python-list


From charleshixsn at earthlink.net  Fri Dec 23 16:56:39 2016
From: charleshixsn at earthlink.net (Charles Hixson)
Date: Fri, 23 Dec 2016 13:56:39 -0800
Subject: Multiprocessing queues receiving from wrong process
Message-ID: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>

I was looking to avoid using a upd connection to transfer messages 
between processes, so I thought I'd use multiprocessing (which I expect 
would be faster), but...I sure would appreciate an explanation of this 
problem.

When I run the code (below) instead of messages receiving messages from 
the correct process I get:
(where the first number of a line identifies the index assigned to the 
process.)

waiting for completion
1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
1=Process-2 received: at time 0.000885: Process-5 says Hi to 4

##    Test multiprocessing queues
import    multiprocessing    as    mp
import    time

from    multiprocessing    import    Process
from    multiprocessing    import    Queue
from    queue                    import    Empty
from    queue                    import    Full
from    random                import    random


class TestMPQ:
     """ Class doc """

     def __init__ (self, ndx):
         """ Class initialiser """
         self.name    =    mp.current_process().name
         self.ndx    =    ndx

     def    sendHi (self):
         for i in range(5):
             if i != self.ndx:
                 qs[i].put ("{} says Hi to {}".format(self.name, self.ndx))

     def    processHi (self):
         while (True):
             time.sleep(random() + 0.001)
             try:
                 msg    =    qs[self.ndx].get_nowait()
                 print ("{}={} received: {}".format(self.ndx, self.name, 
msg) )
             except    Empty:
                 break
             except Exception as ex:
                 print ("Exception: ", repr(ex))
                 break

def    procHandler (ndx, qs):
     p    =    TestMPQ(ndx)
     p.sendHi()
     p.processHi()

if "__main__" == __name__:
     qs    =    []
     for i in range(5):
         qs.append(Queue())
     ps    =    []
     for i in range(5):
         ps.append(Process(target = procHandler, args = (i, qs) ) )
         ps[i].start()
     print ("waiting for completion")
     for i in range(5):
         ps[i].join()




From thinkwelldesigns at gmail.com  Fri Dec 23 17:25:16 2016
From: thinkwelldesigns at gmail.com (thinkwell)
Date: Fri, 23 Dec 2016 14:25:16 -0800 (PST)
Subject: Python 3.6 on Centos 6
Message-ID: <8c1a0cbc-250d-4534-b3b4-a832372e89e5@googlegroups.com>

I'm trying to build Python 3.6 on Centos 6, and am successful in doing so, except for the sqlite3 library. I started with a brand new install of Centos 6 and installed devtoolset-2 to build with a newer compiler. But whether with default compiler or 4.82, I get the following errors when building the sqlite3 module. sqlite & sqlite-devel are installed.

[code]
building '_sqlite3' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/cache.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/connection.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/cursor.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o
/tmp/Python-3.6.0/Modules/_sqlite/cursor.c: In function ?_pysqlite_query_execute?:
/tmp/Python-3.6.0/Modules/_sqlite/cursor.c:517:5: warning: implicit declaration of function ?sqlite3_stmt_readonly? [-Wimplicit-function-declaration]
     if (self->connection->begin_statement && !sqlite3_stmt_readonly(self->statement->st) && !self->statement->is_ddl) {
     ^
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/microprotocols.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/module.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/row.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/statement.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/util.c -o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o
gcc -pthread -shared -Wl,--rpath=/usr/lib build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o -L. -L/usr/local/lib -lsqlite3 -lpython3.6m -o build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so
warning: building with the bundled copy of libffi is deprecated on this platform.  It will not be distributed with Python 3.7
*** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly
[/code]

Any help for me in getting python 3.6 built on Centos 6?


From python at mrabarnett.plus.com  Fri Dec 23 18:19:23 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 23 Dec 2016 23:19:23 +0000
Subject: Multiprocessing queues receiving from wrong process
In-Reply-To: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>
References: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>
Message-ID: <940fb9fc-a5aa-02e2-8bd6-43c37c3b0663@mrabarnett.plus.com>

On 2016-12-23 21:56, Charles Hixson wrote:
> I was looking to avoid using a upd connection to transfer messages
> between processes, so I thought I'd use multiprocessing (which I expect
> would be faster), but...I sure would appreciate an explanation of this
> problem.
>
> When I run the code (below) instead of messages receiving messages from
> the correct process I get:
> (where the first number of a line identifies the index assigned to the
> process.)
>
> waiting for completion
> 1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
> 0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
> 0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
> 4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
> 0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
> 0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
> 3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
> 3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
> 4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
> 2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
> 4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
> 4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
> 1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
> 3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
> 2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
> 2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
> 2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
> 3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
> 1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
> 1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
>
I don't see a bug, but perhaps that's because it's not clear to me what 
you expected to see.

> ##    Test multiprocessing queues
> import    multiprocessing    as    mp
> import    time
>
> from    multiprocessing    import    Process
> from    multiprocessing    import    Queue
> from    queue                    import    Empty
> from    queue                    import    Full
> from    random                import    random
>
>
> class TestMPQ:
>      """ Class doc """
>
>      def __init__ (self, ndx):
>          """ Class initialiser """
>          self.name    =    mp.current_process().name
>          self.ndx    =    ndx
>
>      def    sendHi (self):
>          for i in range(5):
>              if i != self.ndx:
>                  qs[i].put ("{} says Hi to {}".format(self.name, self.ndx))
>
>      def    processHi (self):
>          while (True):
>              time.sleep(random() + 0.001)
>              try:
>                  msg    =    qs[self.ndx].get_nowait()
>                  print ("{}={} received: {}".format(self.ndx, self.name,
> msg) )
>              except    Empty:
>                  break
>              except Exception as ex:
>                  print ("Exception: ", repr(ex))
>                  break
>
> def    procHandler (ndx, qs):
>      p    =    TestMPQ(ndx)
>      p.sendHi()
>      p.processHi()
>
> if "__main__" == __name__:
>      qs    =    []
>      for i in range(5):
>          qs.append(Queue())
>      ps    =    []
>      for i in range(5):
>          ps.append(Process(target = procHandler, args = (i, qs) ) )
>          ps[i].start()
>      print ("waiting for completion")
>      for i in range(5):
>          ps[i].join()
>
>



From __peter__ at web.de  Fri Dec 23 18:24:46 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 24 Dec 2016 00:24:46 +0100
Subject: data frame
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
 <235400868.1750803.1482525599916@mail.yahoo.com>
 <o3k3o7$onp$1@blaine.gmane.org>
 <1242451632.1794626.1482530144997@mail.yahoo.com>
Message-ID: <o3kbnq$e73$1@blaine.gmane.org>

Val Krem via Python-list wrote:

> Here is the first few lines of the data
> 
> 
> s1.csv
> size,w1,h1
> 512,214,26
> 123,250,34
> 234,124,25
> 334,213,43

Did you put these lines here using copy and paste? The fix below depends on 
the assumption that your data is more like

size, w1, h1
512, 214, 26
123, 250, 34
...

>     a=pd.read_csv("s1.csv", skipinitialspace=True).keys()

You should use the keys() method call for diagnosis only. The final script 
that might work if your problem is actually space after the commas is

import pandas as  pd

a = pd.read_csv("s1.csv", skipinitialspace=True)
a["test"] = a["h1"] + a["w1"]
print(a)




From no.email at nospam.invalid  Fri Dec 23 19:20:18 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Fri, 23 Dec 2016 16:20:18 -0800
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
Message-ID: <87k2aqyy7x.fsf@nightsong.com>

> "Salted hashing (or just hashing) with BLAKE2 or any other
> general-purpose cryptographic hash function, such as SHA-256, is not
> suitable for hashing passwords. See BLAKE2 FAQ for more information."
>
> I propose to ignore this warning. I feel that, for my purposes, the
> above procedure is adequate.
>
> Does all this sound reasonable?

The basic problem is those functions are fast enough to make dictionary
attacks feasible.  The preferred password hashing function these days is
Argon2, which has some tunable security parameters:

  https://en.wikipedia.org/wiki/Argon2

Also in use are bcrypt and later scrypt (I think there are wikipedia
articles about both).

What is it that you are trying to secure?  If it's something important,
set up 2-factor authentication (such as TOTP) and encourage your users
to use it.  There are cheap hardware tokens and cost-free smartphone
apps that implement it on the user side.  Linotp.org has a free
implementation for the server side, though I haven't looked at it
closely or tried it yet.


From charleshixsn at earthlink.net  Fri Dec 23 20:17:54 2016
From: charleshixsn at earthlink.net (Charles Hixson)
Date: Fri, 23 Dec 2016 17:17:54 -0800
Subject: Multiprocessing queues receiving from wrong process
In-Reply-To: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>
References: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>
Message-ID: <512ef2a5-8662-fcb8-6bef-8ceae0c267a0@earthlink.net>



On 12/23/2016 01:56 PM, Charles Hixson wrote:
> I was looking to avoid using a upd connection to transfer messages 
> between processes, so I thought I'd use multiprocessing (which I 
> expect would be faster), but...I sure would appreciate an explanation 
> of this problem.
>
> When I run the code (below) instead of messages receiving messages 
> from the correct process I get:
> (where the first number of a line identifies the index assigned to the 
> process.)
>
> waiting for completion
The receiving process should be the one sent to.

          receiving sending                   sent to
   ndx process name                                    process 
name          ndx
> 1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
> 0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
> 0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
> 4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
> 0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
> 0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
> 3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
> 3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
> 4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
> 2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
> 4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
> 4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
> 1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
> 3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
> 2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
> 2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
> 2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
> 3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
> 1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
> 1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
>
> ##    Test multiprocessing queues
> import    multiprocessing    as    mp
> import    time
>
> from    multiprocessing    import    Process
> from    multiprocessing    import    Queue
> from    queue                    import    Empty
> from    queue                    import    Full
> from    random                import    random
>
>
> class TestMPQ:
>     """ Class doc """
>
>     def __init__ (self, ndx):
>         """ Class initialiser """
>         self.name    =    mp.current_process().name
>         self.ndx    =    ndx
>
>     def    sendHi (self):
>         for i in range(5):
>             if i != self.ndx:
>                 qs[i].put ("{} says Hi to {}".format(self.name, 
> self.ndx))
>
>     def    processHi (self):
>         while (True):
>             time.sleep(random() + 0.001)
>             try:
>                 msg    =    qs[self.ndx].get_nowait()
>                 print ("{}={} received: {}".format(self.ndx, 
> self.name, msg) )
>             except    Empty:
>                 break
>             except Exception as ex:
>                 print ("Exception: ", repr(ex))
>                 break
>
> def    procHandler (ndx, qs):
>     p    =    TestMPQ(ndx)
>     p.sendHi()
>     p.processHi()
>
> if "__main__" == __name__:
>     qs    =    []
>     for i in range(5):
>         qs.append(Queue())
>     ps    =    []
>     for i in range(5):
>         ps.append(Process(target = procHandler, args = (i, qs) ) )
>         ps[i].start()
>     print ("waiting for completion")
>     for i in range(5):
>         ps[i].join()
>
>



From rosuav at gmail.com  Fri Dec 23 20:31:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 12:31:52 +1100
Subject: Another security question
In-Reply-To: <87k2aqyy7x.fsf@nightsong.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
Message-ID: <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>

On Sat, Dec 24, 2016 at 11:20 AM, Paul Rubin <no.email at nospam.invalid> wrote:
> The basic problem is those functions are fast enough to make dictionary
> attacks feasible.  The preferred password hashing function these days is
> Argon2, which has some tunable security parameters:

Solution: Don't use dictionary-attackable passwords. Here's a quick
and stupid password verifier that uses MD5:

import os, base64, hashlib, timeit

def encrypt_password(pw):
    salt = base64.b64encode(os.urandom(3))
    return salt + b":" + hashlib.md5(salt + pw).hexdigest().encode("ascii")

def verify_password(pw, enc):
    salt, hash = enc.split(b":")
    return hash == hashlib.md5(salt + pw).hexdigest().encode("ascii")

crackme = encrypt_password(b"elven-their-cure-planning")
assert not verify_password(b"figure-dagger-personal-excited", crackme)
assert verify_password(b"elven-their-cure-planning", crackme)

print(timeit.repeat('verify_password(b"figure-dagger-personal-excited",
crackme)', globals=globals()))
# [0.9219889058731496, 0.9255746062844992, 0.9231259850785136,
0.9204203351400793, 0.9239354520104825]


Okay, so that's about as insecure as salted hashes can get, right? I
brute-forced a million of them a second, one core. Now, suppose you
know for sure that I used that format of password (four words, all
lower-case, separated by hyphens), and you have a copy of my 2000-word
dictionary from which I select words. It'll still take you 185 days to
exhaustively try every possible password. So on average, it'll take
you a day of searching with 93 computers to crack this kind of
password. And that's just using four characters of salt and a single
iteration of MD5. That's borderline infeasible, without needing any
sort of top-notch hashing system. Switch to werkzeug's functions
and... well, I had to tweak timeit.repeat, because it did about two
*thousand* per second. And that's with default parameters. You're
going to need five hundred times as much processing to crack those
things.

Use XKCD 936 passwords. Even if someone has your entire password
generation system, they're still pretty secure. If anything, they'll
be even harder to crack, because of the many variations you could use
(capitalization, separators, precise list of words), not to mention
that you can easily add a fifth word to make them even more secure.
(But only if you're paranoid. Memorizing a fifth word is harder.) It
works.

ChrisA


From steve+python at pearwood.info  Fri Dec 23 20:32:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 24 Dec 2016 12:32:07 +1100
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
Message-ID: <585dd01a$0$1584$c3e8da3$5496439d@news.astraweb.com>

On Sat, 24 Dec 2016 11:20 am, Paul Rubin wrote:

> What is it that you are trying to secure? ?If it's something important,
> set up 2-factor authentication (such as TOTP) and encourage your users
> to use it.


You say that as if two-factor auth was a panacea.

That's the sort of thinking that leads to:

https://www.schneier.com/blog/archives/2009/09/hacking_two-fac.html
https://www.schneier.com/blog/archives/2005/10/scandinavian_at_1.html
http://resources.infosecinstitute.com/two-factor-authentication/
http://www.securityweek.com/two-factor-authentication-bypassed-simple-attacks


not to mention the abomination of "one factor authentication, twice", like
that used by the Australian government unified web portal. To log in, you
have to provide something you know (username and password), plus something
else you know (answer to a low-security question like "what was your
mother's maiden name?").



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Fri Dec 23 20:39:21 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 12:39:21 +1100
Subject: Another security question
In-Reply-To: <585dd01a$0$1584$c3e8da3$5496439d@news.astraweb.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <585dd01a$0$1584$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpSCeS2kfRE=vS2F2HGwZFGUosESPDFDXBOKYsoMbZ_1A@mail.gmail.com>

On Sat, Dec 24, 2016 at 12:32 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> not to mention the abomination of "one factor authentication, twice", like
> that used by the Australian government unified web portal. To log in, you
> have to provide something you know (username and password), plus something
> else you know (answer to a low-security question like "what was your
> mother's maiden name?").

My mother's maiden name was here-campaigns-your-really. My first pet
was expensive-items-know-thats. The street I grew up on was
sorry-days-standard-just. My date of birth is
notes-wondering-laptop-think.

:D

ChrisA


From python at mrabarnett.plus.com  Fri Dec 23 20:50:07 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 24 Dec 2016 01:50:07 +0000
Subject: Multiprocessing queues receiving from wrong process
In-Reply-To: <512ef2a5-8662-fcb8-6bef-8ceae0c267a0@earthlink.net>
References: <da21f15f-941c-9bcd-0d23-07c317b381da@earthlink.net>
 <512ef2a5-8662-fcb8-6bef-8ceae0c267a0@earthlink.net>
Message-ID: <22074d00-544a-ba0d-9630-9e077688abd6@mrabarnett.plus.com>

On 2016-12-24 01:17, Charles Hixson wrote:
>
>
> On 12/23/2016 01:56 PM, Charles Hixson wrote:
>> I was looking to avoid using a upd connection to transfer messages
>> between processes, so I thought I'd use multiprocessing (which I
>> expect would be faster), but...I sure would appreciate an explanation
>> of this problem.
>>
>> When I run the code (below) instead of messages receiving messages
>> from the correct process I get:
>> (where the first number of a line identifies the index assigned to the
>> process.)
>>
>> waiting for completion
> The receiving process should be the one sent to.
>
>           receiving sending                   sent to
>    ndx process name                                    process
> name          ndx
>> 1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
>> 0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
>> 0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
>> 4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
>> 0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
>> 0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
>> 3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
>> 3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
>> 4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
>> 2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
>> 4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
>> 4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
>> 1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
>> 3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
>> 2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
>> 2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
>> 2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
>> 3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
>> 1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
>> 1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
>>
>> ##    Test multiprocessing queues
>> import    multiprocessing    as    mp
>> import    time
>>
>> from    multiprocessing    import    Process
>> from    multiprocessing    import    Queue
>> from    queue                    import    Empty
>> from    queue                    import    Full
>> from    random                import    random
>>
>>
>> class TestMPQ:
>>     """ Class doc """
>>
>>     def __init__ (self, ndx):
>>         """ Class initialiser """
>>         self.name    =    mp.current_process().name
>>         self.ndx    =    ndx
>>
>>     def    sendHi (self):
>>         for i in range(5):
>>             if i != self.ndx:
>>                 qs[i].put ("{} says Hi to {}".format(self.name,
>> self.ndx))
>>
"self.ndx" is the sender's (my) index, "i" is the receiver's (your 
index), so the message you're building is:

     <my name> says Hi to <my index>

which you then put in the receiver's (your) queue.

>>     def    processHi (self):
>>         while (True):
>>             time.sleep(random() + 0.001)
>>             try:
>>                 msg    =    qs[self.ndx].get_nowait()
>>                 print ("{}={} received: {}".format(self.ndx,
>> self.name, msg) )
>>             except    Empty:
>>                 break
>>             except Exception as ex:
>>                 print ("Exception: ", repr(ex))
>>                 break
>>
>> def    procHandler (ndx, qs):
>>     p    =    TestMPQ(ndx)
>>     p.sendHi()
>>     p.processHi()
>>
>> if "__main__" == __name__:
>>     qs    =    []
>>     for i in range(5):
>>         qs.append(Queue())
>>     ps    =    []
>>     for i in range(5):
>>         ps.append(Process(target = procHandler, args = (i, qs) ) )
>>         ps[i].start()
>>     print ("waiting for completion")
>>     for i in range(5):
>>         ps[i].join()
>>
>>
>



From no.email at nospam.invalid  Sat Dec 24 02:18:45 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Fri, 23 Dec 2016 23:18:45 -0800
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
Message-ID: <87fuldztey.fsf@nightsong.com>

Chris Angelico <rosuav at gmail.com> writes:
> Solution: Don't use dictionary-attackable passwords. 

If you allow people to choose their own passwords, they'll too-often
pick dictionary-attackable ones; or even if they choose difficult ones,
they'll use them in more than one place, and eventually the weakest of
those places will eventually leak it.  At that point it can be tried
against whatever other hashes the attacker collected.

The -real- right thing to do is use a secret-keyed hash function like
HMAC-whatever, but that gives you a chicken-and-egg problem of how to
get the secret into the system and prevent it from escaping, when you've
postulated that the hashed passwords might escape.  That's doable
through hardware approaches like external crypto modules, TPM, secure
enclaves in the CPU, etc.; but none of that is very widely deployed at
the moment, and it brings its own unattractiveness.


From rosuav at gmail.com  Sat Dec 24 02:38:47 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 18:38:47 +1100
Subject: Another security question
In-Reply-To: <87fuldztey.fsf@nightsong.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
 <87fuldztey.fsf@nightsong.com>
Message-ID: <CAPTjJmqsZ2-EUD=y9fdqi-q2BAs7C4tzZ3Ue+WxAEq22YVYgEQ@mail.gmail.com>

On Sat, Dec 24, 2016 at 6:18 PM, Paul Rubin <no.email at nospam.invalid> wrote:
> Chris Angelico <rosuav at gmail.com> writes:
>> Solution: Don't use dictionary-attackable passwords.
>
> If you allow people to choose their own passwords, they'll too-often
> pick dictionary-attackable ones; or even if they choose difficult ones,
> they'll use them in more than one place, and eventually the weakest of
> those places will eventually leak it.  At that point it can be tried
> against whatever other hashes the attacker collected.

Correct. However, weak passwords are ultimately the user's
responsibility, where the hashing is the server's responsibility. The
one thing that you _can_ do as server admin is to make appropriate
recommendations, including that if you have one of those "password is
weak" warnings, make sure it favours length over apparent alphabet.

ChrisA


From marko at pacujo.net  Sat Dec 24 02:58:34 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Sat, 24 Dec 2016 09:58:34 +0200
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <585dd01a$0$1584$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87pokhahcl.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> https://www.schneier.com/blog/archives/2005/10/scandinavian_at_1.html

    EDITED TO ADD: Here's a related story. The Bank of New Zealand
    suspended Internet banking because of phishing concerns. Now there's
    a company that is taking the threat seriously.

That's the trouble both in nature and in business: in order to make a
living, you must expose yourself to predators.


Marko


From no.email at nospam.invalid  Sat Dec 24 03:04:53 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Sat, 24 Dec 2016 00:04:53 -0800
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <585dd01a$0$1584$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87bmw1zra2.fsf@nightsong.com>

Steve D'Aprano <steve+python at pearwood.info> writes:
> You say that as if two-factor auth was a panacea.

Of course it's not a panacea, but it helps quite a lot.

> That's the sort of thinking that leads to: ...

Beyond that, web browsers are the new Microsoft Windows with all of its
security holes and bloat and upgrade treadmill.  Depending on the
application, delivering it over the web might simply never be ok.


From no.email at nospam.invalid  Sat Dec 24 03:08:16 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Sat, 24 Dec 2016 00:08:16 -0800
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
 <87fuldztey.fsf@nightsong.com>
 <CAPTjJmqsZ2-EUD=y9fdqi-q2BAs7C4tzZ3Ue+WxAEq22YVYgEQ@mail.gmail.com>
 <mailman.251.1482565560.2322.python-list@python.org>
Message-ID: <877f6pzr4f.fsf@nightsong.com>

Chris Angelico <rosuav at gmail.com> writes:
> Correct. However, weak passwords are ultimately the user's
> responsibility, where the hashing is the server's responsibility.

No, really, the users are part of the system and therefore the system
designer must take the expected behavior of actual users into account.
The idea is to prevent breaches, not to allow them as long as the blame
can be shifted to someone else.


From rosuav at gmail.com  Sat Dec 24 03:29:02 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 24 Dec 2016 19:29:02 +1100
Subject: Another security question
In-Reply-To: <877f6pzr4f.fsf@nightsong.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
 <87fuldztey.fsf@nightsong.com>
 <CAPTjJmqsZ2-EUD=y9fdqi-q2BAs7C4tzZ3Ue+WxAEq22YVYgEQ@mail.gmail.com>
 <mailman.251.1482565560.2322.python-list@python.org>
 <877f6pzr4f.fsf@nightsong.com>
Message-ID: <CAPTjJmrKanoqQcZYOMHOKEqmYsvDUA4RWyKF4TDpyxREF4j_9A@mail.gmail.com>

On Sat, Dec 24, 2016 at 7:08 PM, Paul Rubin <no.email at nospam.invalid> wrote:
> Chris Angelico <rosuav at gmail.com> writes:
>> Correct. However, weak passwords are ultimately the user's
>> responsibility, where the hashing is the server's responsibility.
>
> No, really, the users are part of the system and therefore the system
> designer must take the expected behavior of actual users into account.
> The idea is to prevent breaches, not to allow them as long as the blame
> can be shifted to someone else.

I agree, but that's why I said "ultimately". As an end user of a
system, I have no control over the hashing used, and lots of control
over the password I use; as a sysadmin, I have lots of control over
the hashing, and very little on passwords. I could enforce a minimum
password length, but I can't prevent password reuse, and I can't do
much about the other forms of weak passwords.

ChrisA


From frank at chagford.com  Sat Dec 24 03:43:32 2016
From: frank at chagford.com (Frank Millman)
Date: Sat, 24 Dec 2016 10:43:32 +0200
Subject: Another security question
In-Reply-To: <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o3j0jt$3pc$1@blaine.gmane.org>
 <mailman.223.1482491346.2322.python-list@python.org>
 <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o3lcfk$pah$1@blaine.gmane.org>

"Steve D'Aprano"  wrote in message 
news:585d57d5$0$1587$c3e8da3$5496439d at news.astraweb.com...
>
> There is a stdlib PBKDF2. If you want to avoid third-party dependencies, 
> use that.
>
> https://docs.python.org/3.4/library/hashlib.html#hashlib.pbkdf2_hmac
>

Thanks for the pointer.

>From the docs - 15.1.3. Key derivation -
"The number of iterations should be chosen based on the hash algorithm and 
computing power. As of 2013, at least 100,000 iterations of SHA-256 are 
suggested."

So FWIW, this is what I have come up with -

from hashlib import pbkdf2_hmac as kdf
from secrets import token_bytes
from json import loads, dumps

def gen_password(pwd):
    hash_name = 'sha256'
    salt = token_bytes(16)
    iterations = 100000
    dk = kdf(hash_name, pwd.encode('utf-8'), salt, iterations)
    return dumps([hash_name, salt.hex(), iterations, dk.hex()])

def chk_password(pwd_hash, pwd):
    hash_name, salt, iterations, dk = loads(pwd_hash)
    return (kdf(hash_name, pwd.encode('utf-8'), bytes.fromhex(salt), 
iterations)
        == bytes.fromhex(dk))

pwd = 'this is my secret passphrase'

pwd_hash = gen_password(pwd)
print(pwd_hash)
print(chk_password(pwd_hash, pwd))

["sha256", "2cd1150b98dab7219136c8deceda00e3", 100000, 
"6301857d79554c3e2035fc779e4903f098ba2df36536028b72952426a5773f0a"]
True

I know that 'rolling your own' is a no-no when it comes to security. I don't 
know whether this falls into that category or not, but I will run with it 
for now.

Frank




From frank at chagford.com  Sat Dec 24 04:01:02 2016
From: frank at chagford.com (Frank Millman)
Date: Sat, 24 Dec 2016 11:01:02 +0200
Subject: Another security question
In-Reply-To: <o3lcfk$pah$1@blaine.gmane.org>
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <585d009f$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <o3j0jt$3pc$1@blaine.gmane.org>
 <mailman.223.1482491346.2322.python-list@python.org>
 <585d57d5$0$1587$c3e8da3$5496439d@news.astraweb.com>
 <o3lcfk$pah$1@blaine.gmane.org>
Message-ID: <o3ldgd$o9n$1@blaine.gmane.org>

"Frank Millman"  wrote in message news:o3lcfk$pah$1 at blaine.gmane.org...

By the way, I have realised how I ended up getting sidetracked by Blake2 in 
the first place.

If you call up the online documentation for Python3.6 and select modules>h> 
hashlib, it takes you straight to
    15.2. hashlib ? BLAKE2 hash functions?

It should take you to
    15.1. hashlib ? Secure hashes and message digests

I think I should raise a 'documentation issue' to report this.

Frank




From no.email at nospam.invalid  Sat Dec 24 05:12:28 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Sat, 24 Dec 2016 02:12:28 -0800
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
 <87fuldztey.fsf@nightsong.com>
 <CAPTjJmqsZ2-EUD=y9fdqi-q2BAs7C4tzZ3Ue+WxAEq22YVYgEQ@mail.gmail.com>
 <mailman.251.1482565560.2322.python-list@python.org>
 <877f6pzr4f.fsf@nightsong.com>
 <CAPTjJmrKanoqQcZYOMHOKEqmYsvDUA4RWyKF4TDpyxREF4j_9A@mail.gmail.com>
 <mailman.252.1482568145.2322.python-list@python.org>
Message-ID: <8737hdzldf.fsf@nightsong.com>

Chris Angelico <rosuav at gmail.com> writes:
> as a sysadmin, I have lots of control over the hashing, and very
> little on passwords. I could enforce a minimum password length, but I
> can't prevent password reuse, and I can't do much about the other
> forms of weak passwords.

Right, 2FA helps with re-use, and difficult hashes like Argon2 help
against dictionary attacks.  Whether 2FA is worth the hassle to depends
on what's being secured.  You can also assign system-generated passwords
rather than having people choose their own.  It's ok for them to write
down the system-generated passwords as long as they keep the paper in a
safe place (similar to how they would carry cash).  There's a Schneier
blog post about that someplace.


From raulmaqueda6398 at gmail.com  Sat Dec 24 05:27:05 2016
From: raulmaqueda6398 at gmail.com (raulmaqueda6398 at gmail.com)
Date: Sat, 24 Dec 2016 02:27:05 -0800 (PST)
Subject: I need a lot of help...
Message-ID: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>

I do not know how to do this exercise, does anyone help me?

Define the matrix_range (m) function that returns the range of an array calculated by the Gaussian method.

It should work with any number of rows and columns. No punctuation will be given to deliveries that do not respect this requirement.


Operating example:

>>> matrix_range ([[1,0,0], [0,1,0], [0,0,1]])
3


From raulmaqueda6398 at gmail.com  Sat Dec 24 05:30:10 2016
From: raulmaqueda6398 at gmail.com (raulmaqueda6398 at gmail.com)
Date: Sat, 24 Dec 2016 02:30:10 -0800 (PST)
Subject: I need a lot of help...
In-Reply-To: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
Message-ID: <c4a3848a-98b2-450f-94e6-39a35088b316@googlegroups.com>

El s?bado, 24 de diciembre de 2016, 11:27:16 (UTC+1), raulmaq... at gmail.com  escribi?:
> I do not know how to do this exercise, does anyone help me?
> 
> Define the matrix_range (m) function that returns the range of an array calculated by the Gaussian method.
> 
> It should work with any number of rows and columns. No punctuation will be given to deliveries that do not respect this requirement.
> 
> 
> Operating example:
> 
> >>> matrix_range ([[1,0,0], [0,1,0], [0,0,1]])
> 3

I need to do it in python


From __peter__ at web.de  Sat Dec 24 06:15:52 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 24 Dec 2016 12:15:52 +0100
Subject: I need a lot of help...
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
Message-ID: <o3lld5$skv$1@blaine.gmane.org>

raulmaqueda6398 at gmail.com wrote:

> I do not know how to do this exercise, does anyone help me?
> 
> Define the matrix_range (m) function that returns the range of an array
> calculated by the Gaussian method.
> 
> It should work with any number of rows and columns. No punctuation will be
> given to deliveries that do not respect this requirement.
> 
> 
> Operating example:
> 
>>>> matrix_range ([[1,0,0], [0,1,0], [0,0,1]])
> 3

If this were not a homework problem you'd use numpy.linalg.matrix_range().
In your case, however, your teacher wants you to implement the algorithm 
yourself.

Do you know how to calculate the matrix range with pen and paper? 

Write down the steps in plain english and then try to implement them one 
after another in Python. Once you have some code and run into a problem you 
have something to show us, and we'll happily give you the hints necessary to 
fix it.



From kushal at locationd.net  Sat Dec 24 07:29:29 2016
From: kushal at locationd.net (Kushal Kumaran)
Date: Sat, 24 Dec 2016 17:59:29 +0530
Subject: Python 3.6 on Centos 6
In-Reply-To: <8c1a0cbc-250d-4534-b3b4-a832372e89e5@googlegroups.com>
 (thinkwell's message of "Fri, 23 Dec 2016 14:25:16 -0800 (PST)")
References: <8c1a0cbc-250d-4534-b3b4-a832372e89e5@googlegroups.com>
Message-ID: <87vau9o6hi.fsf@locationd.net>

thinkwell <thinkwelldesigns at gmail.com> writes:

> I'm trying to build Python 3.6 on Centos 6, and am successful in doing
> so, except for the sqlite3 library. I started with a brand new install
> of Centos 6 and installed devtoolset-2 to build with a newer
> compiler. But whether with default compiler or 4.82, I get the
> following errors when building the sqlite3 module. sqlite &
> sqlite-devel are installed.
>
> [code]
> building '_sqlite3' extension
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/cache.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/connection.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/cursor.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o
> /tmp/Python-3.6.0/Modules/_sqlite/cursor.c: In function ?_pysqlite_query_execute?:
> /tmp/Python-3.6.0/Modules/_sqlite/cursor.c:517:5: warning: implicit declaration of function ?sqlite3_stmt_readonly? [-Wimplicit-function-declaration]
>      if (self->connection->begin_statement && !sqlite3_stmt_readonly(self->statement->st) && !self->statement->is_ddl) {
>      ^
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/microprotocols.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/module.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/row.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/statement.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o
> gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3"
> -IModules/_sqlite -I/usr/include -I./Include -I. -I/usr/local/include
> -I/tmp/Python-3.6.0/Include -I/tmp/Python-3.6.0 -c
> /tmp/Python-3.6.0/Modules/_sqlite/util.c -o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o
> gcc -pthread -shared -Wl,--rpath=/usr/lib
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o
> build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o
> -L. -L/usr/local/lib -lsqlite3 -lpython3.6m -o
> build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so
> warning: building with the bundled copy of libffi is deprecated on this platform.  It will not be distributed with Python 3.7
> *** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly
> [/code]
>
> Any help for me in getting python 3.6 built on Centos 6?

What version of the sqlite3 libraries do you have?  See
https://github.com/ghaering/pysqlite/issues/85 for example.

--
regards,
kushal


From valkrem at yahoo.com  Sat Dec 24 10:37:52 2016
From: valkrem at yahoo.com (Val Krem)
Date: Sat, 24 Dec 2016 15:37:52 +0000 (UTC)
Subject: data frame
In-Reply-To: <o3kbnq$e73$1@blaine.gmane.org>
References: <235400868.1750803.1482525599916.ref@mail.yahoo.com>
 <235400868.1750803.1482525599916@mail.yahoo.com>
 <o3k3o7$onp$1@blaine.gmane.org>
 <1242451632.1794626.1482530144997@mail.yahoo.com>
 <o3kbnq$e73$1@blaine.gmane.org>
Message-ID: <2121712730.2166279.1482593872367@mail.yahoo.com>

Thank you Peter and  Christ.
It is was a white space and the fix fixed it.
Many thanks.




On Friday, December 23, 2016 5:26 PM, Peter Otten <__peter__ at web.de> wrote:
Val Krem via Python-list wrote:

> Here is the first few lines of the data
> 
> 
> s1.csv
> size,w1,h1
> 512,214,26
> 123,250,34
> 234,124,25
> 334,213,43

Did you put these lines here using copy and paste? The fix below depends on 
the assumption that your data is more like

size, w1, h1
512, 214, 26
123, 250, 34
...

>     a=pd.read_csv("s1.csv", skipinitialspace=True).keys()

You should use the keys() method call for diagnosis only. The final script 
that might work if your problem is actually space after the commas is

import pandas as  pd

a = pd.read_csv("s1.csv", skipinitialspace=True)
a["test"] = a["h1"] + a["w1"]
print(a)


-- 
https://mail.python.org/mailman/listinfo/python-list


From philb at philb.ca  Sat Dec 24 12:18:35 2016
From: philb at philb.ca (Phil Boutros)
Date: Sat, 24 Dec 2016 17:18:35 -0000 (UTC)
Subject: I need a lot of help...
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
Message-ID: <slrno5tbi3.drf.philb@ah61.is-a-geek.com>

raulmaqueda6398 at gmail.com <raulmaqueda6398 at gmail.com> wrote:
> I do not know how to do this exercise, does anyone help me?

    If you have a specific, precise question, that's one thing.
Otherwise:

    http://lmgtfy.com/?q=Academic+Dishonesty

Phil
-- 
AH#61  Wolf#14  BS#89  bus#1  CCB#1  SENS  KOTC#4
hilb at philb.ca  http://philb.ca


From thinkwelldesigns at gmail.com  Sat Dec 24 14:47:22 2016
From: thinkwelldesigns at gmail.com (thinkwell)
Date: Sat, 24 Dec 2016 11:47:22 -0800 (PST)
Subject: Python 3.6 on Centos 6
In-Reply-To: <mailman.256.1482583220.2322.python-list@python.org>
References: <8c1a0cbc-250d-4534-b3b4-a832372e89e5@googlegroups.com>
 <87vau9o6hi.fsf@locationd.net>
 <mailman.256.1482583220.2322.python-list@python.org>
Message-ID: <0a405608-5dd6-4e4a-bd24-6c458fcf0f00@googlegroups.com>


> What version of the sqlite3 libraries do you have?  See
> https://github.com/ghaering/pysqlite/issues/85 for example.
> 
> --
> regards,
> kushal

I was using the system default for Centos 6 which is 3.6.20. I loaded a parallel version and now got 3.6 to build. Thanks!


From pkpearson at nowhere.invalid  Sat Dec 24 18:49:18 2016
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 24 Dec 2016 23:49:18 GMT
Subject: I need a lot of help...
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
Message-ID: <ec8fruF32jjU1@mid.individual.net>

On Sat, 24 Dec 2016 02:27:05 -0800 (PST), raulmaqueda6398 at gmail.com wrote:
> I do not know how to do this exercise, does anyone help me?
> 
> Define the matrix_range (m) function that returns the range of an
> array calculated by the Gaussian method.
>
> It should work with any number of rows and columns. No punctuation
> will be given to deliveries that do not respect this requirement.
>
> Operating example:
>
>>>> matrix_range ([[1,0,0], [0,1,0], [0,0,1]])
> 3

I don't know any definition of "matrix range" that fits this description.
Is it possible that someone means "rank"?

-- 
To email me, substitute nowhere->runbox, invalid->com.


From no.email at nospam.invalid  Sat Dec 24 22:08:11 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Sat, 24 Dec 2016 19:08:11 -0800
Subject: I need a lot of help...
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
 <ec8fruF32jjU1@mid.individual.net>
Message-ID: <87y3z4yack.fsf@nightsong.com>

Peter Pearson <pkpearson at nowhere.invalid> writes:
> I don't know any definition of "matrix range" that fits this description.
> Is it possible that someone means "rank"?

Yes, the rank is the dimension of the range unless I'm mistaken.  I
think rank is what was meant.

To find the rank with Gaussian elimination, I guess you could
diagonalize the matrix then count the number of nonzero elements.  Maybe
there's a better way.


From sunnycemetery at gmail.com  Sun Dec 25 00:50:16 2016
From: sunnycemetery at gmail.com (Grady Martin)
Date: Sun, 25 Dec 2016 00:50:16 -0500
Subject: UTF-8 Encoding Error
In-Reply-To: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
Message-ID: <20161225055016.GI16063@creation>

On 2016?12?22? 22?38?, subhabangalore at gmail.com wrote:
>I am getting the error:
>UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15: invalid start byte

The following is a reflex of mine, whenever I encounter Python 2 Unicode errors:

import sys
reload(sys)
sys.setdefaultencoding('utf8')

A relevant Stack Exchange thread awaits you here:

http://stackoverflow.com/a/21190382/2230956


From __peter__ at web.de  Sun Dec 25 03:13:50 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 25 Dec 2016 09:13:50 +0100
Subject: I need a lot of help...
References: <3c83c0d2-1151-47e0-a7d8-a9b2425686af@googlegroups.com>
 <o3lld5$skv$1@blaine.gmane.org>
Message-ID: <o3nv43$b7i$1@blaine.gmane.org>

Peter Otten wrote:

> raulmaqueda6398 at gmail.com wrote:
> 
>> I do not know how to do this exercise, does anyone help me?
>> 
>> Define the matrix_range (m) function that returns the range of an array
>> calculated by the Gaussian method.
>> 
>> It should work with any number of rows and columns. No punctuation will
>> be given to deliveries that do not respect this requirement.
>> 
>> 
>> Operating example:
>> 
>>>>> matrix_range ([[1,0,0], [0,1,0], [0,0,1]])
>> 3
> 
> If this were not a homework problem you'd use numpy.linalg.matrix_range().

Of course Peter Pearson's hint applies to my answer. The above function 
doesn't exist, it should be numpy.linalg.matrix_rank()

>>> numpy.linalg.matrix_rank([[1,0,0], [0,1,0], [0,0,1]])
3

which should not be confused with numpy.rank() that just returns the number 
of dimensions:

>>> numpy.rank([[1,0,0], [0,1,0], [0,0,1]])
2
>>> numpy.rank([[[[[[]]]]]])
6

> In your case, however, your teacher wants you to implement the algorithm
> yourself.
> 
> Do you know how to calculate the matrix range with pen and paper?
> 
> Write down the steps in plain english and then try to implement them one
> after another in Python. Once you have some code and run into a problem
> you have something to show us, and we'll happily give you the hints
> necessary to fix it.




From steve+python at pearwood.info  Sun Dec 25 08:44:29 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 26 Dec 2016 00:44:29 +1100
Subject: Another security question
References: <o3itmj$t42$1@blaine.gmane.org>
 <mailman.220.1482488369.2322.python-list@python.org>
 <87k2aqyy7x.fsf@nightsong.com>
 <CAPTjJmq8KSVuyvbX2TNEN_L5gj-OTt+7iRoEvvHKaijZGKpOpQ@mail.gmail.com>
 <mailman.248.1482543121.2322.python-list@python.org>
 <87fuldztey.fsf@nightsong.com>
 <CAPTjJmqsZ2-EUD=y9fdqi-q2BAs7C4tzZ3Ue+WxAEq22YVYgEQ@mail.gmail.com>
 <mailman.251.1482565560.2322.python-list@python.org>
Message-ID: <585fcd40$0$1601$c3e8da3$5496439d@news.astraweb.com>

On Sat, 24 Dec 2016 06:38 pm, Chris Angelico wrote:

> weak passwords are ultimately the user's
> responsibility


I suppose that's true, in the same way that not getting sewerage into the
drinking water supply is also ultimately the user's responsibility.

You forget that weak passwords don't just hurt the user who choose the weak
passwords. If I break into your system, I get the opportunity to steal your
identity, which not only hurts you, but also those I steal from using your
identity. I can use your account to send spam, which hurts everyone. I can
use you as a springboard to attack others, to launch ransomware attacks or
shutdown the electricity grid[1] or DOS people I don't like.

Poor security eventually hurts everyone.

I think that, eventually, one of two things will happen:

- Our entire computing infrastructure (the web, email, the IOTs, banking
systems, etc) will collapse under the accumulated weight of zero day
attacks, malware, ransomware, cyber warfare and 24/7 surveillance by both
the state and corporations. The IOT is an especially bad idea:

http://www.geekculture.com/joyoftech/joyarchives/2340.html


- Or governments realise that computing security (including privacy) needs
to be treated as a public health measure.

We're already aware of the virus metaphor when it comes to malicious code.
(It's more than just a metaphor -- one can argue, correctly I think, that
self-replicating code is the same kind of thing whether it is interpreted
by a Word macro, compiled machine code, or DNA.) We also need to think of
personal data as toxic pollution: 

https://www.schneier.com/blog/archives/2016/03/data_is_a_toxic.html

https://www.schneier.com/blog/archives/2008/01/data_as_polluti.html

We need to be thinking about security vulnerabilities as a health issue.
That includes the backdoors more and more governments will want us to
install, under the false claim of protecting us from terrorists/
paedophiles/whatever villain is being demonised this year. Exploitable
software needs to be treated the same as building a sewer system that
empties directly into the city's drinking water supply.

It's *everybody's* problem when somebody can hack into your vulnerable
system. That's the ultimate externality. 

But of course, unfortunately, we know what most governments and corporations
and even individuals think about pollution and toxic waste. "If it saves me
5 seconds, or earns me $1, I don't care how many billions in damages it
does to others."

Merry Christmas.





"My light switch is currently downloading a software update from the
Internet so I can't turn my lights off. What. A. Time. To. Be. Alive."
https://twitter.com/TweetsByTSD/status/655297659381661696




[1] If any country is foolish enough to put control of the electricity grid
on the Internet. Of course nobody would do that. Right?



-- 
Steve



From gvm2121 at gmail.com  Sun Dec 25 17:07:10 2016
From: gvm2121 at gmail.com (Gonzalo V)
Date: Sun, 25 Dec 2016 19:07:10 -0300
Subject: UTF-8 Encoding Error
In-Reply-To: <20161225055016.GI16063@creation>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
 <20161225055016.GI16063@creation>
Message-ID: <CAF40P_3c48POB24cmdhR6e5gTCPjq1i7ROMKfwKyZHGT48Smtw@mail.gmail.com>

Try utf-8-sig
El 25 dic. 2016 2:57 AM, "Grady Martin" <sunnycemetery at gmail.com> escribi?:

> On 2016?12?22? 22?38?, subhabangalore at gmail.com wrote:
>
>> I am getting the error:
>> UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15:
>> invalid start byte
>>
>
> The following is a reflex of mine, whenever I encounter Python 2 Unicode
> errors:
>
> import sys
> reload(sys)
> sys.setdefaultencoding('utf8')
>
> A relevant Stack Exchange thread awaits you here:
>
> http://stackoverflow.com/a/21190382/2230956
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From alec.taylor6 at gmail.com  Mon Dec 26 02:05:34 2016
From: alec.taylor6 at gmail.com (Alec Taylor)
Date: Mon, 26 Dec 2016 18:05:34 +1100
Subject: logger.info / logger.error with logger.addHandler - how to split
 streams?
Message-ID: <CAO+9iGeudrQqF-TnMCPhpk4WKLFq-z41oc5gfU4sjD_cKKnMfg@mail.gmail.com>

So I'm putting .info in one StringIO and .error in another StringIO.

How do I stop them from both being put into both?

Code: http://ideone.com/Nj6Asz


From jumppanen.jussi at gmail.com  Mon Dec 26 02:27:08 2016
From: jumppanen.jussi at gmail.com (jumppanen.jussi at gmail.com)
Date: Sun, 25 Dec 2016 23:27:08 -0800 (PST)
Subject: Python 3.6 Embedded
Message-ID: <5be30e1e-873c-4ae9-ab33-33884b8316af@googlegroups.com>

I'm using the python3.6 DLL to embed Python 3 inside a Windows application.

Here is some background into how Python is being used.

1. Firstly the same basic code worked fine with the earlier Python 2.7 version.

2. The code is structured as follows:

    Windows Executable
        +
        |
        + Scripting DLL
              +
              |
              + Pthyhon 3.6 DLL

In other words the executable talks to the Scripting DLL and the scripting DLL then talks to the Python DLL.

3. To run a script the following flow control is used:

    a. Application loads the Scripting DLL
        i.  The Scripting DLL calls Py_Initialize found in the Python DLL

    b. Application run the script file using the Scripting DLL
        i.  The Scripting DLL calls PyRun_SimpleFileEx in the Python DLL to run the script

    c. Unload the scripting DLL
        i.  The Scripting DLL calls Py_FinalizeEx in the Python DLL clean up

Symptoms of the Problem
-----------------------
The problem can be replicated by a script that contains nothing but a single import statement.

Using this test script the script can be run and re-run any number of times:

    import sys

Using this test script the script will run the first time but fail every subsequent time:

    import socket

The difference in the two import statements is the former is a built-in module while the later is using a dynamically loaded module.

The Cause of the Problem
------------------------
I tracked down the cause of the problem to the fact the python36.dll unloads just fine for the first case but fails to unloaded for the second case.

So when the application loads and unloads the Scripting DLL the python36.dll remains loaded.

Then when the python36.dll is used the second time it now contains pointers to stale data and crashes on the second run.

The Py_FinalizeEx document found here suggests it should stop all interpreters: https://docs.python.org/dev/c-api/init.html

Anyone have an idea know what might be going wrong?

What code is holding on to the Python DLL?


From stefan_ml at behnel.de  Mon Dec 26 10:55:52 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 26 Dec 2016 16:55:52 +0100
Subject: windows utf8 & lxml
In-Reply-To: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
References: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
Message-ID: <o3rei3$sl6$1@blaine.gmane.org>

Hi!

Sayth Renshaw schrieb am 20.12.2016 um 12:53:
> I have been trying to get a script to work on windows that works on mint. The key blocker has been utf8 errors, most of which I have solved.
> 
> Now however the last error I am trying to overcome, the solution appears to be to use the .decode('windows-1252') to correct an ascii error.
> 
> I am using lxml to read my content and decode is not supported are there any known ways to read with lxml and fix unicode faults?
> 
> The key part of my script is 
> 
>         for content in roots:
>             utf8_parser = etree.XMLParser(encoding='utf-8')
>             fix_ascii = utf8_parser.decode('windows-1252')

This looks rather broken. Are you sure this is what your code looks like,
or did just you type this into your email while trying to strip down your
actual code into a simpler example?


>             mytree = etree.fromstring(
>                 content.read().encode('utf-8'), parser=fix_ascii)

Note that lxml can parse from Unicode, so once you have decoded your data,
you can just pass it into the parser as is, e.g.

    mytree = etree.fromstring(content.decode('windows-1252'))

This is not something I'd encourage since it requires a bit of back and
forth encoding internally and is rather memory inefficient, but if your
decoding is non-trivial, this might still be a viable approach.


> Without the added .decode my code looks like
> 
>         for content in roots:
>             utf8_parser = etree.XMLParser(encoding='utf-8')
>             mytree = etree.fromstring(
>                 content.read().encode('utf-8'), parser=utf8_parser)
> 
> However doing it in such a fashion returns this error:
> 
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Same thing as above: I don't see how this error message matches the code
you show here. The exception you get might be a Python 2.x problem in the
first place.

Stefan



From g.starck at gmail.com  Mon Dec 26 10:57:21 2016
From: g.starck at gmail.com (gst)
Date: Mon, 26 Dec 2016 07:57:21 -0800 (PST)
Subject: logger.info / logger.error with logger.addHandler - how to split
 streams?
In-Reply-To: <mailman.263.1482766472.2322.python-list@python.org>
References: <CAO+9iGeudrQqF-TnMCPhpk4WKLFq-z41oc5gfU4sjD_cKKnMfg@mail.gmail.com>
 <mailman.263.1482766472.2322.python-list@python.org>
Message-ID: <83439ac0-e4e1-4b37-abd5-e89bb00f5800@googlegroups.com>

Le lundi 26 d?cembre 2016 10:34:48 UTC-5, Alec Taylor a ?crit?:
> So I'm putting .info in one StringIO and .error in another StringIO.
> 
> How do I stop them from both being put into both?
> 
> Code: http://ideone.com/Nj6Asz


Hi,

it's doable with filter on the handlers:


def exact_filter(level):
    def filter(record):
        return level == record.levelno
    filter.filter = filter
    return filter

stdout_stream.addFilter(exact_filter(logging.INFO))
stderr_stream.addFilter(exact_filter(logging.ERROR))


From steve+python at pearwood.info  Tue Dec 27 05:46:35 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 27 Dec 2016 21:46:35 +1100
Subject: windows utf8 & lxml
References: <e0e4d749-f472-4277-adbd-7c8693512cc7@googlegroups.com>
Message-ID: <5862468d$0$1595$c3e8da3$5496439d@news.astraweb.com>

On Tue, 20 Dec 2016 10:53 pm, Sayth Renshaw wrote:

> content.read().encode('utf-8'), parser=utf8_parser)
> 
> However doing it in such a fashion returns this error:
> 
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0:
> invalid start byte


That tells you that the XML file you have is not actually UTF-8.

You have a file that begins with a byte 0xFF. That is invalid UTF-8. No
valid UTF-8 string contains the byte 0xFF.

https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences

So you need to consider:

- Are you sure that the input file is intended to be UTF-8? How was it
created? 

- Is the second byte 0xFE? If so, that suggests that you actually have
UTF-16 with a byte-order mark.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From hpj at urpla.net  Tue Dec 27 08:37:45 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Tue, 27 Dec 2016 14:37:45 +0100
Subject: ctypes, memory mapped files and context manager
Message-ID: <7976395.RisPf5L5KU@xrated>

Hi,

I'm using $subjects combination successfully in a project for 
creating/iterating over huge binary files (> 5GB) with impressive performance, 
while resource usage keeps pretty low, all with plain Python3 code. Nice!

Environment: (Python 3.4.5, Linux 4.8.14, openSUSE/x86_64, NFS4 and XFS 
filesystems)

The idea is: map a ctypes structure onto the file at a certain offset, act on 
the structure, and release the mapping. The latter is necessary for keeping 
the mmap file properly resizable and closable (due to the nature of mmaps and 
Python's posix implementation thereof). Hence, a context manager serves us 
well (in theory). 

Here's some code excerpt:

class cstructmap:
    def __init__(self, cstruct, mm, offset = 0):
        self._cstruct = cstruct
        self._mm = mm
        self._offset = offset
        self._csinst = None

    def __enter__(self):
        # resize the mmap (and backing file), if structure exceeds mmap size
        # mmap size must be aligned to mmap.PAGESIZE
        cssize = ctypes.sizeof(self._cstruct)
        if self._offset + cssize > self._mm.size():
            newsize = align(self._offset + cssize, mmap.PAGESIZE)
            self._mm.resize(newsize)
        self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
        return self._csinst

    def __exit__(self, exc_type, exc_value, exc_traceback):
        # free all references into mmap
        del self._csinst
        self._csinst = None


def work():
    with cstructmap(ItemHeader, self._mm, self._offset) as ih:
        ih.identifier = ItemHeader.Identifier
        ih.length = ItemHeaderSize + datasize
        
    blktype = ctypes.c_char * datasize
    with cstructmap(blktype, self._mm, self._offset) as blk:
        blk.raw = data


In practice, this results in:

Traceback (most recent call last):
  File "ctypes_mmap_ctx.py", line 146, in <module>
    mf.add_data(data)
  File "ctypes_mmap_ctx.py", line 113, in add_data
    with cstructmap(blktype, self._mm, self._offset) as blk:
  File "ctypes_mmap_ctx.py", line 42, in __enter__
    self._mm.resize(newsize)
BufferError: mmap can't resize with extant buffers exported.

The issue: when creating a mapping via context manager, we assign a local 
variable (with ..), that keep existing in the local context, even when the 
manager context was left. This keeps a reference on the ctypes mapped area 
alive, even if we try everything to destroy it in __exit__. We have to del the 
with var manually.

Now, I want to get rid of the ugly any error prone del statements.

What is needed, is a ctypes operation, that removes the mapping actively, and 
that could be added to the __exit__ part of the context manager.

Full working code example: 
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239

The script creates a memory mapped file in the current directory named 
"mapfile". When started without arguments, it copies itself into this file, 
until 10 * mmap.PAGESIZE growth is reached (or it errored out before..).

IF you change NOPROB to True, it will actively destruct the context manager 
vars, and should work as advertized.

Any ideas are much appreciated.

Thanks in advance,
Pete



From skip.montanaro at gmail.com  Tue Dec 27 10:05:37 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Tue, 27 Dec 2016 07:05:37 -0800 (PST)
Subject: encoding="utf8" ignored when parsing XML
Message-ID: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>

I am trying to parse some XML which doesn't specify an encoding (Python 2.7.12 via Anaconda on RH Linux), so it barfs when it encounters non-ASCII data. No great surprise there, but I'm having trouble getting it to use another encoding. First, I tried specifying the encoding when opening the file:

f = io.open(fname, encoding="utf8")
root = xml.etree.ElementTree.parse(f).getroot()

but that had no effect. Then, when calling xml.etree.ElementTree.parse I included an XMLParser object:

parser = xml.etree.ElementTree.XMLParser(encoding="utf8")
root = xml.etree.ElementTree.parse(f, parser=parser).getroot()

Same-o, same-o:

unicode error 'ascii' codec can't encode characters in position 1113-1116: ordinal not in range(128)

So, why does it continue to insist on using an ASCII codec? My locale's preferred encoding is:

>>> locale.getpreferredencoding()
'ANSI_X3.4-1968'

which I presume is the official way to spell "ascii".

The chardetect command (part of the chardet package) tells me it looks like utf8 with high confidence:

% chardetect < ~/tmp/trash
<stdin>: utf-8 with confidence 0.99

I took a look at the code, and tracked the encoding I specified all the way down to the creation of the expat parser. What am I missing?

Skip


From __peter__ at web.de  Tue Dec 27 10:25:51 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 27 Dec 2016 16:25:51 +0100
Subject: encoding="utf8" ignored when parsing XML
References: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>
Message-ID: <o3u15r$62a$1@blaine.gmane.org>

Skip Montanaro wrote:

> I am trying to parse some XML which doesn't specify an encoding (Python
> 2.7.12 via Anaconda on RH Linux), so it barfs when it encounters non-ASCII
> data. No great surprise there, but I'm having trouble getting it to use
> another encoding. First, I tried specifying the encoding when opening the
> file:
> 
> f = io.open(fname, encoding="utf8")
> root = xml.etree.ElementTree.parse(f).getroot()
> 
> but that had no effect. 

Isn't UTF-8 the default?

Try opening the file in binary mode then:

with io.open(fname, "rb") as f:
    root = xml.tree.ElementTree.parse(f).getroot()




From skip.montanaro at gmail.com  Tue Dec 27 10:47:16 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Tue, 27 Dec 2016 09:47:16 -0600
Subject: encoding="utf8" ignored when parsing XML
In-Reply-To: <o3u15r$62a$1@blaine.gmane.org>
References: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>
 <o3u15r$62a$1@blaine.gmane.org>
Message-ID: <CANc-5UxKk24_RShAZqJ_do6M4GrQoWaw35vzb8zBRa71ZRDXNg@mail.gmail.com>

Peter> Isn't UTF-8 the default?

Apparently not. I believe in my reading it said that it used whatever
locale.getpreferredencoding() returned. That's problematic when you
live in a country that thinks ASCII is everything. Personally, I think
UTF-8 should be the default, but that train's long left the station,
at least for Python 2.x.

> Try opening the file in binary mode then:
>
> with io.open(fname, "rb") as f:
>     root = xml.tree.ElementTree.parse(f).getroot()

Thanks, that worked. Would appreciate an explanation of why binary
mode was necessary. It would seem that since the file contents are
text, just in a non-ASCII encoding, that specifying the encoding when
opening the file should do the trick.

Skip


From __peter__ at web.de  Tue Dec 27 11:10:41 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 27 Dec 2016 17:10:41 +0100
Subject: encoding="utf8" ignored when parsing XML
References: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>
 <o3u15r$62a$1@blaine.gmane.org>
 <CANc-5UxKk24_RShAZqJ_do6M4GrQoWaw35vzb8zBRa71ZRDXNg@mail.gmail.com>
Message-ID: <o3u3ps$b9s$1@blaine.gmane.org>

Skip Montanaro wrote:

> Peter> Isn't UTF-8 the default?
> 
> Apparently not. 

Sorry, I meant the default for XML.

> I believe in my reading it said that it used whatever
> locale.getpreferredencoding() returned. That's problematic when you
> live in a country that thinks ASCII is everything. Personally, I think
> UTF-8 should be the default, but that train's long left the station,
> at least for Python 2.x.
> 
>> Try opening the file in binary mode then:
>>
>> with io.open(fname, "rb") as f:
>>     root = xml.tree.ElementTree.parse(f).getroot()
> 
> Thanks, that worked. Would appreciate an explanation of why binary
> mode was necessary. It would seem that since the file contents are
> text, just in a non-ASCII encoding, that specifying the encoding when
> opening the file should do the trick.
> 
> Skip

My tentative explanation would be: If you open the file as text it will be 
successfully decoded, i. e.

io.open(fname, encoding="UTF-8").read()

works, but to go back to the bytes that the XML parser needs the "preferred 
encoding", in your case ASCII, will be used. 

Since there are non-ascii characters you get a UnicodeEncodeError.




From __peter__ at web.de  Tue Dec 27 11:19:34 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 27 Dec 2016 17:19:34 +0100
Subject: encoding="utf8" ignored when parsing XML
References: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>
 <o3u15r$62a$1@blaine.gmane.org>
 <CANc-5UxKk24_RShAZqJ_do6M4GrQoWaw35vzb8zBRa71ZRDXNg@mail.gmail.com>
 <o3u3ps$b9s$1@blaine.gmane.org>
Message-ID: <o3u4ah$e5r$1@blaine.gmane.org>

Peter Otten wrote:

> works, but to go back to the bytes that the XML parser needs the
> "preferred encoding", in your case ASCII, will be used.

Correction: it's probably sys.getdefaultencoding() rather than 
locale.getdefaultencoding(). So all systems with a sane configuration will 
behave the same way as yours.



From 1991manish.kumar at gmail.com  Tue Dec 27 14:46:52 2016
From: 1991manish.kumar at gmail.com (1991manish.kumar at gmail.com)
Date: Tue, 27 Dec 2016 11:46:52 -0800 (PST)
Subject: Parse a Wireshark pcap file
In-Reply-To: <mailman.853.1358904722.2939.python-list@python.org>
References: <mailman.853.1358904722.2939.python-list@python.org>
Message-ID: <236debf2-d81d-4d54-83a7-7cbb50184853@googlegroups.com>


I have a pcap file, I want to parse that file & fetch some information like Timestamp, Packet Size, Source/Dest IP Address, Source/Dest Port, Source/ Dest MAC address.

I am trying this in Django.

other that Source/ Dest Port details, I am able to fetch everything.
Please tell me how I can get port details from pcap file.

This is my python code: https://github.com/manishkk/pcap-parser/blob/master/webapp/views.py

Thanking you in advance

On Wednesday, January 23, 2013 at 2:32:00 AM UTC+1, KDawg44 wrote:
> Is there a way to parse out a wireshark pcap file and extract key value pairs from the data? ?I am illustrated a sniff of some traffic and why it needs utilize HTTPS instead of HTTP but I was hoping to run the pcap through a python script and just output some interesting key value pairs....?
> 
> 
> 
> Thanks for your help.
> 
> 
> Kevin



From diego at vida.com  Tue Dec 27 15:15:01 2016
From: diego at vida.com (Diego Vela)
Date: Tue, 27 Dec 2016 12:15:01 -0800
Subject: Mock object bug with assert_not_called
Message-ID: <CAKJZ8rJCYVtXbaFh3xm7BcyWzV+4gKJtXnyZW1S8gcgQVo0uwA@mail.gmail.com>

Dear all,

>From reading the documentation it seemed like this is the place to post a
bug.  If not please let me know where is the proper place to do so.

Bug:
For mock objects created using the @patch decorator the following methods
are inconsistent.
assert_not_called,
assert_called_with

Steps
Create a mock object using the @patch decorator for a function.
call a function that that calls the mocked function.
check assert_not_called and assert_called with.

Thank you for your time.

-- 
Diego Vela


From __peter__ at web.de  Tue Dec 27 15:39:51 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 27 Dec 2016 21:39:51 +0100
Subject: ctypes, memory mapped files and context manager
References: <7976395.RisPf5L5KU@xrated>
Message-ID: <o3ujii$1kv$1@blaine.gmane.org>

Hans-Peter Jansen wrote:

> Hi,
> 
> I'm using $subjects combination successfully in a project for
> creating/iterating over huge binary files (> 5GB) with impressive
> performance, while resource usage keeps pretty low, all with plain Python3
> code. Nice!
> 
> Environment: (Python 3.4.5, Linux 4.8.14, openSUSE/x86_64, NFS4 and XFS
> filesystems)
> 
> The idea is: map a ctypes structure onto the file at a certain offset, act
> on the structure, and release the mapping. The latter is necessary for
> keeping the mmap file properly resizable and closable (due to the nature
> of mmaps and Python's posix implementation thereof). Hence, a context
> manager serves us well (in theory).
> 
> Here's some code excerpt:
> 
> class cstructmap:
>     def __init__(self, cstruct, mm, offset = 0):
>         self._cstruct = cstruct
>         self._mm = mm
>         self._offset = offset
>         self._csinst = None
> 
>     def __enter__(self):
>         # resize the mmap (and backing file), if structure exceeds mmap
>         # size mmap size must be aligned to mmap.PAGESIZE
>         cssize = ctypes.sizeof(self._cstruct)
>         if self._offset + cssize > self._mm.size():
>             newsize = align(self._offset + cssize, mmap.PAGESIZE)
>             self._mm.resize(newsize)
>         self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
>         return self._csinst

Here you give away a reference to the ctypes.BigEndianStructure. That means 
you no longer control the lifetime of self._csinst which in turn holds a 
reference to the underlying mmap or whatever it's called. 

There might be a way to release the mmap reference while the wrapper 
structure is still alive, but the cleaner way is probably to not give it 
away in the first place, and create a proxy instead with

          return weakref.proxy(self._csinst)


> 
>     def __exit__(self, exc_type, exc_value, exc_traceback):
>         # free all references into mmap
>         del self._csinst

The line above is redundant. It removes the attribute from the instance 
__dict__ and implicitly decreases its refcount. It does not actually 
physically delete the referenced object. If you remove the del statement the 
line below will still decrease the refcount. 

Make sure you understand this to avoid littering your code with cargo cult 
del-s ;)

>         self._csinst = None
> 
> 
> def work():
>     with cstructmap(ItemHeader, self._mm, self._offset) as ih:
>         ih.identifier = ItemHeader.Identifier
>         ih.length = ItemHeaderSize + datasize
>         
>     blktype = ctypes.c_char * datasize
>     with cstructmap(blktype, self._mm, self._offset) as blk:
>         blk.raw = data
> 
> 
> In practice, this results in:
> 
> Traceback (most recent call last):
>   File "ctypes_mmap_ctx.py", line 146, in <module>
>     mf.add_data(data)
>   File "ctypes_mmap_ctx.py", line 113, in add_data
>     with cstructmap(blktype, self._mm, self._offset) as blk:
>   File "ctypes_mmap_ctx.py", line 42, in __enter__
>     self._mm.resize(newsize)
> BufferError: mmap can't resize with extant buffers exported.
> 
> The issue: when creating a mapping via context manager, we assign a local
> variable (with ..), that keep existing in the local context, even when the
> manager context was left. This keeps a reference on the ctypes mapped area
> alive, even if we try everything to destroy it in __exit__. We have to del
> the with var manually.
> 
> Now, I want to get rid of the ugly any error prone del statements.
> 
> What is needed, is a ctypes operation, that removes the mapping actively,
> and that could be added to the __exit__ part of the context manager.
> 
> Full working code example:
> https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239
> 
> The script creates a memory mapped file in the current directory named
> "mapfile". When started without arguments, it copies itself into this
> file, until 10 * mmap.PAGESIZE growth is reached (or it errored out
> before..).
> 
> IF you change NOPROB to True, it will actively destruct the context
> manager vars, and should work as advertized.
> 
> Any ideas are much appreciated.

You might put some more effort into composing example scripts. Something 
like the script below would have saved me some time...

import ctypes
import mmap

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    yield T.from_buffer(m)

SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1) as a:
    a.foo = 1
with map_struct(m, 2) as b:
    b.foo = 2


> 
> Thanks in advance,
> Pete




From steve+python at pearwood.info  Tue Dec 27 20:40:32 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 28 Dec 2016 12:40:32 +1100
Subject: encoding="utf8" ignored when parsing XML
References: <948af3e7-3b0d-4dda-8955-5cb90c846a76@googlegroups.com>
Message-ID: <58631813$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Wed, 28 Dec 2016 02:05 am, Skip Montanaro wrote:

> I am trying to parse some XML which doesn't specify an encoding (Python
> 2.7.12 via Anaconda on RH Linux), so it barfs when it encounters non-ASCII
> data. No great surprise there, but I'm having trouble getting it to use
> another encoding. First, I tried specifying the encoding when opening the
> file:
> 
> f = io.open(fname, encoding="utf8")
> root = xml.etree.ElementTree.parse(f).getroot()

The documentation for ET.parse is pretty sparse

https://docs.python.org/2/library/xml.etree.elementtree.html#xml.etree.ElementTree.parse


but we can infer that it should take bytes as argument, not Unicode, since
it does its own charset processing. (The optional parser argument takes an
encoding argument which defaults to UTF-8.)

So that means using the built-in open(), or io.open() in binary mode.

You open the file and read in bytes from disk, *decoding* those bytes into a
UTF-8 Unicode string. Then the ET parser tries to decode the Unicode string
into Unicode, which it does by first *encoding* it back to bytes using the
default encoding (namely ASCII), and that's where it blows up.

This particular error is a Python2-ism, since Python2 tries hard to let you
mix byte strings and unicode strings together, hence it will try implicitly
encoding/decoding strings to try to get them to fit together. Python3 does
not do this.

You can easily simulate this error at the REPL:



py> u"?".decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb5' in position
0: ordinal not in range(128)


The give-away is that you're intending to do a *decode* operation but get an
*encode* error. That tells you that Python2 is trying to be helpful :-)

(Remember: Unicode strings encode to bytes, and bytes decode back to
strings.)


You're trying to read bytes from a file on disk and get Unicode strings out:

bytes in file --> XML parser --> Unicode

so that counts as a decoding operation. But you're getting an encoding
error -- that's the smoking gun that suggests a dubious Unicode->bytes
step, using the default encoding (ASCII):

bytes in file --> io.open().read() --> Unicode --> XML Parser --> decode to
bytes using ASCII --> encode back to Unicode using UTF-8

And that suggests that the fix is to open the file without any charset
processing, i.e. use the builtin open() instead of io.open().

bytes in file --> builtin open().read() --> bytes --> XML Parser --> Unicode


I think you can even skip the 'rb' mode part: the real problem is that you
must not feed a Unicode string to the XML parser.



> but that had no effect. Then, when calling xml.etree.ElementTree.parse I
> included an XMLParser object:
> 
> parser = xml.etree.ElementTree.XMLParser(encoding="utf8")
> root = xml.etree.ElementTree.parse(f, parser=parser).getroot()

That's the default, so there's no functional change here. Hence, the same
error.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Dec 27 20:45:27 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 28 Dec 2016 12:45:27 +1100
Subject: Mock object bug with assert_not_called
References: <CAKJZ8rJCYVtXbaFh3xm7BcyWzV+4gKJtXnyZW1S8gcgQVo0uwA@mail.gmail.com>
 <mailman.5.1482870420.2395.python-list@python.org>
Message-ID: <58631939$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Wed, 28 Dec 2016 07:15 am, Diego Vela wrote:

> Dear all,
> 
>>From reading the documentation it seemed like this is the place to post a
> bug.  If not please let me know where is the proper place to do so.

Once you are sure that it truly is a bug, then the proper place is the bug
tracker:

http://bugs.python.org/

This is a good place to ask if you are unsure whether it is or isn't a bug.


> Bug:
> For mock objects created using the @patch decorator the following methods
> are inconsistent.
> assert_not_called,
> assert_called_with
> 
> Steps
> Create a mock object using the @patch decorator for a function.
> call a function that that calls the mocked function.
> check assert_not_called and assert_called with.

Thank you, but this bug report can be improved.

(1) What results do you expect? What results do you get?

(2) What version of Python are you using?

(3) Code is better than English. Can you supply a *minimal* (as small and
simple as possible) example that shows this bug?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From guohaochuan at gmail.com  Wed Dec 28 05:03:15 2016
From: guohaochuan at gmail.com (Haochuan Guo)
Date: Wed, 28 Dec 2016 10:03:15 +0000
Subject: Garbage collection problem with generators
In-Reply-To: <CAF=uXRWep5YTjUGxCB2fFO-V4kL6PC5ymadP_em68mYXybZaVg@mail.gmail.com>
References: <CAF=uXRWep5YTjUGxCB2fFO-V4kL6PC5ymadP_em68mYXybZaVg@mail.gmail.com>
Message-ID: <CAF=uXRW9a+72gE+Wm7JNwM1uqxfu=_0nMV0+Ar-MfP8J3eFxVA@mail.gmail.com>

Anyone? The script to reproduce this problem is in:

https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef

On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo <guohaochuan at gmail.com> wrote:

> Hi, everyone
>
> I'm building a http long polling client for our company's discovery
> service and something weird happened in the following code:
>
> ```python
> while True:
>     try:
>         r = requests.get("url", stream=True, timeout=3)
>         for data in r.iter_lines():
>             processing_data...
>     except TimeoutException:
>         time.sleep(10)
> ```
>
> When I deliberately times out the request and then check the connections
> with `lsof -p process`, I discover that there are *two active connections*(ESTABLISH)
> instead of one. After digging around, it turns out it might not be the
> problem with `requests` at all, but gc related to generators.
>
> So I write this script to demonstrate the problem:
>
> https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
>
> Function `A.a` will return a generator which will raise an exception. And
> in function `b`, I'm building new a new instance of `A` and iterate over
> the exception-raising generator. In the exception handler, I'll close the
> generator, delete it, delete the `A` instance, call `gc.collect()` and do
> the whole process all over again.
>
> There's another greenlet checking the `A` instances by using
> `gc.get_objects()`. It turns out there are always two `A` instances.
>
> This is reproducible with python2.7, but not in python3.5. I've also tried
> with `thread` instead of `gevent`, it still happens. I'm guessing it's
> related to garbage collection of generators.
>
> Did I bump into a python2 bug? Or am I simply wrong about the way to close
> generators...?
>
> Thanks
>


From hpj at urpla.net  Wed Dec 28 05:32:44 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Wed, 28 Dec 2016 11:32:44 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <o3ujii$1kv$1@blaine.gmane.org>
References: <7976395.RisPf5L5KU@xrated> <o3ujii$1kv$1@blaine.gmane.org>
Message-ID: <2388223.lqpfiZhTPD@xrated>

Dear Peter,

thanks for taking valuable time to look into my issue.

It might be related to my distinct silliness, but the problem persists with 
your code as well. Further comments inlined.  

On Dienstag, 27. Dezember 2016 21:39:51 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > 
> >     def __enter__(self):
> >         # resize the mmap (and backing file), if structure exceeds mmap
> >         # size mmap size must be aligned to mmap.PAGESIZE
> >         cssize = ctypes.sizeof(self._cstruct)
> >         
> >         if self._offset + cssize > self._mm.size():
> >             newsize = align(self._offset + cssize, mmap.PAGESIZE)
> >             self._mm.resize(newsize)
> >         
> >         self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
> >         return self._csinst
> 
> Here you give away a reference to the ctypes.BigEndianStructure. That means
> you no longer control the lifetime of self._csinst which in turn holds a
> reference to the underlying mmap or whatever it's called.

Here's the code, based on contextmanager:

@contextmanager
def cstructmap(cstruct, mm, offset = 0):
    # resize the mmap (and backing file), if structure exceeds mmap size
    # mmap size must be aligned to mmap.PAGESIZE
    cssize = ctypes.sizeof(cstruct)
    if offset + cssize > mm.size():
        newsize = align(offset + cssize, mmap.PAGESIZE)
        mm.resize(newsize)
    yield cstruct.from_buffer(mm, offset)

While much more concise, I don't understand, how it should make a difference 
relative to the "with" variable lifetime, when used. 
 
> There might be a way to release the mmap reference while the wrapper
> structure is still alive, but the cleaner way is probably to not give it
> away in the first place, and create a proxy instead with
> 
>           return weakref.proxy(self._csinst)

This fails, as it doesn't keep the reference long enough.

> >     def __exit__(self, exc_type, exc_value, exc_traceback):
> >         # free all references into mmap
> >         del self._csinst
> 
> The line above is redundant. It removes the attribute from the instance
> __dict__ and implicitly decreases its refcount. It does not actually
> physically delete the referenced object. If you remove the del statement the
> line below will still decrease the refcount.
> 
> Make sure you understand this to avoid littering your code with cargo cult
> del-s ;)

Yes, I was aware of this. It was a testing relic, that survived somehow. 
Sorry. Yes, I usually try to avoid cargo cultry in my code. ;)

> > The issue: when creating a mapping via context manager, we assign a local
> > variable (with ..), that keep existing in the local context, even when the
> > manager context was left. This keeps a reference on the ctypes mapped area
> > alive, even if we try everything to destroy it in __exit__. We have to del
> > the with var manually.
> > 
> > Now, I want to get rid of the ugly any error prone del statements.
> > 
> > What is needed, is a ctypes operation, that removes the mapping actively,
> > and that could be added to the __exit__ part of the context manager.

Revised code (including your test code):
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239

> > The script creates a memory mapped file in the current directory named
> > "mapfile". When started without arguments, it copies itself into this
> > file, until 10 * mmap.PAGESIZE growth is reached (or it errored out
> > before..).
> > 
> > IF you change NOPROB to True, it will actively destruct the context
> > manager vars, and should work as advertized.
> > 
> > Any ideas are much appreciated.
> 
> You might put some more effort into composing example scripts. Something
> like the script below would have saved me some time...

I'm very sorry about this. 
 
> import ctypes
> import mmap
> 
> from contextlib import contextmanager
> 
> class T(ctypes.Structure):
>     _fields = [("foo", ctypes.c_uint32)]
> 
> 
> @contextmanager
> def map_struct(m, n):
>     m.resize(n * mmap.PAGESIZE)
>     yield T.from_buffer(m)
> 
> SIZE = mmap.PAGESIZE * 2
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
> 
> with map_struct(m, 1) as a:
>     a.foo = 1
> with map_struct(m, 2) as b:
>     b.foo = 2

Unfortunately, your code behaves exactly like mine:

$> python3 mmap_test.py
Traceback (most recent call last):
  File "mmap_test.py", line 23, in <module>
    with map_struct(m, 2) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test.py", line 12, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.

BTW, Python2 makes a difference in this respect, but my project is Python3 
based. Have you tested this with Python3? It would be interesting to explore 
the reasons of this difference, which is, ?hem, pretty surprising.

Thanks,
Pete


From __peter__ at web.de  Wed Dec 28 07:48:48 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 28 Dec 2016 13:48:48 +0100
Subject: ctypes, memory mapped files and context manager
References: <7976395.RisPf5L5KU@xrated> <o3ujii$1kv$1@blaine.gmane.org>
 <2388223.lqpfiZhTPD@xrated>
Message-ID: <o40cbc$t7o$1@blaine.gmane.org>

Hans-Peter Jansen wrote:

> Dear Peter,
> 
> thanks for taking valuable time to look into my issue.

You're welcome!

> It might be related to my distinct silliness, but the problem persists
> with your code as well. 

Unfortunately I posted the broken toy example rather than the fixed one. 
Here's the latter. Basically you have to keep a reference in the context 
manager (whether you implement it as a class or a generator doesn't matter) 
without giving another reference away to client code:

$ cat mmap_after.py
import ctypes
import mmap
import weakref

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    keep_me = T.from_buffer(m)
    yield weakref.proxy(keep_me)


SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1) as a:
    a.foo = 1
with map_struct(m, 2) as b:
    b.foo = 2
$ python3 mmap_after.py 
$ 


If that doesn't suffice to fix it I'll take another look into your real code 
later.




From rosuav at gmail.com  Wed Dec 28 08:11:59 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 29 Dec 2016 00:11:59 +1100
Subject: Garbage collection problem with generators
In-Reply-To: <CAF=uXRW9a+72gE+Wm7JNwM1uqxfu=_0nMV0+Ar-MfP8J3eFxVA@mail.gmail.com>
References: <CAF=uXRWep5YTjUGxCB2fFO-V4kL6PC5ymadP_em68mYXybZaVg@mail.gmail.com>
 <CAF=uXRW9a+72gE+Wm7JNwM1uqxfu=_0nMV0+Ar-MfP8J3eFxVA@mail.gmail.com>
Message-ID: <CAPTjJmqXci878rre5wizNS9+x0gr1-MrzxeVuySGWKsFML+WLQ@mail.gmail.com>

On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo <guohaochuan at gmail.com> wrote:
> Anyone? The script to reproduce this problem is in:
>
> https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
>
> On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo <guohaochuan at gmail.com> wrote:
>
>> This is reproducible with python2.7, but not in python3.5. I've also tried
>> with `thread` instead of `gevent`, it still happens. I'm guessing it's
>> related to garbage collection of generators.
>>
>> Did I bump into a python2 bug? Or am I simply wrong about the way to close
>> generators...?

(Please don't top-post.)

Maybe the fix is to just use Python 3.5+? :) It probably is to do with
the garbage collection of generators; so you may want to consider
using something very explicit (eg a context manager) to ensure that
you call gen.close().

ChrisA


From guohaochuan at gmail.com  Wed Dec 28 08:14:45 2016
From: guohaochuan at gmail.com (Haochuan Guo)
Date: Wed, 28 Dec 2016 13:14:45 +0000
Subject: Garbage collection problem with generators
In-Reply-To: <CAPTjJmqXci878rre5wizNS9+x0gr1-MrzxeVuySGWKsFML+WLQ@mail.gmail.com>
References: <CAF=uXRWep5YTjUGxCB2fFO-V4kL6PC5ymadP_em68mYXybZaVg@mail.gmail.com>
 <CAF=uXRW9a+72gE+Wm7JNwM1uqxfu=_0nMV0+Ar-MfP8J3eFxVA@mail.gmail.com>
 <CAPTjJmqXci878rre5wizNS9+x0gr1-MrzxeVuySGWKsFML+WLQ@mail.gmail.com>
Message-ID: <CAF=uXRXui38b-vwDR_prpCRnZFFC9FM2U_BHc1ixNhha780apQ@mail.gmail.com>

Sorry about breaking the rule.

I'm just curios about this problem. And I'm using this workaround to prevent
redundant resource creation.

https://gist.githubusercontent.com/wooparadog/16948ca6c8ffb22214bf491a280406da/raw/-


On Wed, Dec 28, 2016 at 9:12 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo <guohaochuan at gmail.com>
> wrote:
> > Anyone? The script to reproduce this problem is in:
> >
> > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
> >
> > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo <guohaochuan at gmail.com>
> wrote:
> >
> >> This is reproducible with python2.7, but not in python3.5. I've also
> tried
> >> with `thread` instead of `gevent`, it still happens. I'm guessing it's
> >> related to garbage collection of generators.
> >>
> >> Did I bump into a python2 bug? Or am I simply wrong about the way to
> close
> >> generators...?
>
> (Please don't top-post.)
>
> Maybe the fix is to just use Python 3.5+? :) It probably is to do with
> the garbage collection of generators; so you may want to consider
> using something very explicit (eg a context manager) to ensure that
> you call gen.close().
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From hpj at urpla.net  Wed Dec 28 09:17:22 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Wed, 28 Dec 2016 15:17:22 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <o40cbc$t7o$1@blaine.gmane.org>
References: <7976395.RisPf5L5KU@xrated> <2388223.lqpfiZhTPD@xrated>
 <o40cbc$t7o$1@blaine.gmane.org>
Message-ID: <3560304.JFQfuKfdL7@xrated>

On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > Dear Peter,
> > 
> > thanks for taking valuable time to look into my issue.
> 
> You're welcome!
> 
> > It might be related to my distinct silliness, but the problem persists
> > with your code as well.
> 
> Unfortunately I posted the broken toy example rather than the fixed one.
> Here's the latter. Basically you have to keep a reference in the context
> manager (whether you implement it as a class or a generator doesn't matter)
> without giving another reference away to client code:
> 
> $ cat mmap_after.py
> import ctypes
> import mmap
> import weakref
> 
> from contextlib import contextmanager
> 
> class T(ctypes.Structure):
>     _fields = [("foo", ctypes.c_uint32)]
> 
> 
> @contextmanager
> def map_struct(m, n):
>     m.resize(n * mmap.PAGESIZE)
>     keep_me = T.from_buffer(m)
>     yield weakref.proxy(keep_me)

Hooray, that did the trick. Great solution, thank you very much!

If you don't mind, I will mention you and your solution at the various places, 
I placed this issue over the last weeks.

You made my day, Peter!

It leaves the question on why is Python2 acting as one would expect related to 
context managers, and Python3 needs this weakref juggling. Maybe something, 
that's worth to be placed in python-dev. What do you think?

Cheers,
Pete


From hpj at urpla.net  Wed Dec 28 10:53:53 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Wed, 28 Dec 2016 16:53:53 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <3560304.JFQfuKfdL7@xrated>
References: <7976395.RisPf5L5KU@xrated> <o40cbc$t7o$1@blaine.gmane.org>
 <3560304.JFQfuKfdL7@xrated>
Message-ID: <9101900.Zuuk7oGeDA@xrated>

On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > Hans-Peter Jansen wrote:
> > > Dear Peter,
> > > 
> > > thanks for taking valuable time to look into my issue.
> > 
> > You're welcome!
> > 
> > > It might be related to my distinct silliness, but the problem persists
> > > with your code as well.
> > 
> > Unfortunately I posted the broken toy example rather than the fixed one.
> > Here's the latter. Basically you have to keep a reference in the context
> > manager (whether you implement it as a class or a generator doesn't
> > matter)
> > without giving another reference away to client code:
> > 
> > $ cat mmap_after.py
> > import ctypes
> > import mmap
> > import weakref
> > 
> > from contextlib import contextmanager
> > 
> > class T(ctypes.Structure):
> >     _fields = [("foo", ctypes.c_uint32)]
> > 
> > @contextmanager
> > 
> > def map_struct(m, n):
> >     m.resize(n * mmap.PAGESIZE)
> >     keep_me = T.from_buffer(m)
> >     yield weakref.proxy(keep_me)
> 
> Hooray, that did the trick. Great solution, thank you very much!

Sorry for bothering you again, Peter, but after applying it to the real 
project, that fails consistently similar to:

# first run, check explicitly disabled
$> python3 ctypes_mmap_ctx.py 
DEBUG: starting offset: 0x10
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: final offset: 0xa1d0

$> python3 ctypes_mmap_ctx.py 
Traceback (most recent call last):
  File "ctypes_mmap_ctx.py", line 163, in <module>
    mf = MapFile(mapfile)
  File "ctypes_mmap_ctx.py", line 109, in __init__
    if bytes(blk) != bytes(rest):
AttributeError: 'c_ubyte_Array_3632' object has no attribute '__bytes__'


The new issue appears in some consistency checking code. The mapfile is 
checked for zeroed out overhang on that line. The weakref proxy isn't behaving 
well at that point. Any idea, what could be going wrong with it?


Updated the gist to demonstrate the issue after switching to weakref.proxy().
(has grown even more code in that process, sorry).

 
It looks like a minor issue, but the code quality is really degraded from 
these ugly del statements.

Pete


From larry.martell at gmail.com  Wed Dec 28 15:14:39 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Wed, 28 Dec 2016 15:14:39 -0500
Subject: sorting strings numerically while dealing with missing values
Message-ID: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>

I have a list containing a list of strings that I want to sort
numerically by one of the fields. I am doing this:

sorted(rows, key=float(itemgetter(sortby)))

Which works fine as long as all the sort keys convert to a float.
Problem is that some are blank or None and those throw an exception.
How can I handle that case and still sort? I'd want the blank or None
fields to come out either at the beginning or end of the sorted list
(not sure what the customer wants for this yet).


From __peter__ at web.de  Wed Dec 28 15:39:20 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 28 Dec 2016 21:39:20 +0100
Subject: sorting strings numerically while dealing with missing values
References: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>
Message-ID: <o417tk$js7$1@blaine.gmane.org>

Larry Martell wrote:

> I have a list containing a list of strings that I want to sort
> numerically by one of the fields. I am doing this:
> 
> sorted(rows, key=float(itemgetter(sortby)))
> 
> Which works fine as long as all the sort keys convert to a float.

No, that cannot work; unless you have redefined float() you 'l get a 
TypeError either here

>>> from operator import itemgetter
>>> float(itemgetter(42))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: float() argument must be a string or a number, not 
'operator.itemgetter'

or here:

>>> sorted("abc", key=3.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'float' object is not callable

> Problem is that some are blank or None and those throw an exception.
> How can I handle that case and still sort? I'd want the blank or None
> fields to come out either at the beginning or end of the sorted list
> (not sure what the customer wants for this yet).

Make the key function return a tuple:

>>> def none_blank_float(value):
...     if value is None: return (0,)
...     if not value: return (1,)
...     return 2, float(value)
... 
>>> sorted(["1", "10", "2", None, "3", "", "5"], key=none_blank_float)
[None, '', '1', '2', '3', '5', '10']




From ian.g.kelly at gmail.com  Wed Dec 28 15:43:41 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 28 Dec 2016 14:43:41 -0600
Subject: sorting strings numerically while dealing with missing values
In-Reply-To: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>
References: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>
Message-ID: <CALwzid==a0NPxSzik2zA-McKfkgHFNtA6wNPNssGskc4rMmnvg@mail.gmail.com>

On Wed, Dec 28, 2016 at 2:14 PM, Larry Martell <larry.martell at gmail.com> wrote:
>
> I have a list containing a list of strings that I want to sort
> numerically by one of the fields. I am doing this:
>
> sorted(rows, key=float(itemgetter(sortby)))

I'm guessing that you left out a lambda here since the key argument
takes a function.

> Which works fine as long as all the sort keys convert to a float.
> Problem is that some are blank or None and those throw an exception.
> How can I handle that case and still sort? I'd want the blank or None
> fields to come out either at the beginning or end of the sorted list
> (not sure what the customer wants for this yet).


def sort_key(sortby, none_first=False):
    def key(row):
        try:
            value = float(row[sortby])
        except ValueError:
            value = None
        return ((value is None) != none_first, value)
    return key

sorted(rows, key=sort_key(4, none_first=True))


From ian.g.kelly at gmail.com  Wed Dec 28 15:46:48 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 28 Dec 2016 14:46:48 -0600
Subject: sorting strings numerically while dealing with missing values
In-Reply-To: <CALwzid==a0NPxSzik2zA-McKfkgHFNtA6wNPNssGskc4rMmnvg@mail.gmail.com>
References: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>
 <CALwzid==a0NPxSzik2zA-McKfkgHFNtA6wNPNssGskc4rMmnvg@mail.gmail.com>
Message-ID: <CALwzid=gf-E1oXJyjc1LVFJxR8U1QequVZoOi9JKcn1-KZc7oQ@mail.gmail.com>

On Wed, Dec 28, 2016 at 2:43 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> On Wed, Dec 28, 2016 at 2:14 PM, Larry Martell <larry.martell at gmail.com> wrote:
>>
>> I have a list containing a list of strings that I want to sort
>> numerically by one of the fields. I am doing this:
>>
>> sorted(rows, key=float(itemgetter(sortby)))
>
> I'm guessing that you left out a lambda here since the key argument
> takes a function.
>
>> Which works fine as long as all the sort keys convert to a float.
>> Problem is that some are blank or None and those throw an exception.
>> How can I handle that case and still sort? I'd want the blank or None
>> fields to come out either at the beginning or end of the sorted list
>> (not sure what the customer wants for this yet).
>
>
> def sort_key(sortby, none_first=False):
>     def key(row):
>         try:
>             value = float(row[sortby])
>         except ValueError:
>             value = None
>         return ((value is None) != none_first, value)
>     return key
>
> sorted(rows, key=sort_key(4, none_first=True))

Actually that doesn't quite work because None < None is unorderable in
Python 3. Maybe replace the inner return with:

        return ((value is None) != none_first, value or 0.0)


From __peter__ at web.de  Wed Dec 28 15:58:38 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 28 Dec 2016 21:58:38 +0100
Subject: ctypes, memory mapped files and context manager
References: <7976395.RisPf5L5KU@xrated> <2388223.lqpfiZhTPD@xrated>
 <o40cbc$t7o$1@blaine.gmane.org> <3560304.JFQfuKfdL7@xrated>
Message-ID: <o4191q$vum$1@blaine.gmane.org>

Hans-Peter Jansen wrote:

> On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
>> Hans-Peter Jansen wrote:
>> > Dear Peter,
>> > 
>> > thanks for taking valuable time to look into my issue.
>> 
>> You're welcome!
>> 
>> > It might be related to my distinct silliness, but the problem persists
>> > with your code as well.
>> 
>> Unfortunately I posted the broken toy example rather than the fixed one.
>> Here's the latter. Basically you have to keep a reference in the context
>> manager (whether you implement it as a class or a generator doesn't
>> matter) without giving another reference away to client code:
>> 
>> $ cat mmap_after.py
>> import ctypes
>> import mmap
>> import weakref
>> 
>> from contextlib import contextmanager
>> 
>> class T(ctypes.Structure):
>>     _fields = [("foo", ctypes.c_uint32)]
>> 
>> 
>> @contextmanager
>> def map_struct(m, n):
>>     m.resize(n * mmap.PAGESIZE)
>>     keep_me = T.from_buffer(m)
>>     yield weakref.proxy(keep_me)
> 
> Hooray, that did the trick. Great solution, thank you very much!
> 
> If you don't mind, I will mention you and your solution at the various
> places, I placed this issue over the last weeks.
> 
> You made my day, Peter!
> 
> It leaves the question on why is Python2 acting as one would expect
> related to context managers, and Python3 needs this weakref juggling.
> Maybe something, that's worth to be placed in python-dev. What do you
> think?

Well, given

$ cat mmap_resize.py
import ctypes
import mmap

class T(ctypes.Structure):
    _fields_ = [("foo", ctypes.c_uint32)]

SIZE = 2 * mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)

m = mmap.mmap(f.fileno(), SIZE)

a = T.from_buffer(m, mmap.PAGESIZE)
m.resize(mmap.PAGESIZE)
a.foo = 42
$ python3 mmap_resize.py
Traceback (most recent call last):
  File "mmap_resize.py", line 15, in <module>
    m.resize(mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
$ python2.7 mmap_resize.py
Segmentation fault

do you really prefer the Python 2 behaviour?



From motoom at xs4all.nl  Wed Dec 28 16:54:35 2016
From: motoom at xs4all.nl (Michiel Overtoom)
Date: Wed, 28 Dec 2016 22:54:35 +0100
Subject: Parse a Wireshark pcap file
In-Reply-To: <236debf2-d81d-4d54-83a7-7cbb50184853@googlegroups.com>
References: <mailman.853.1358904722.2939.python-list@python.org>
 <236debf2-d81d-4d54-83a7-7cbb50184853@googlegroups.com>
Message-ID: <6F4767E6-E6EE-4FED-8B08-28CCBFF6F43D@xs4all.nl>


> On 2016-12-27, at 20:46, 1991manish.kumar at gmail.com wrote:
> 
> I have a pcap file, I want to parse that file & fetch some information like Timestamp, Packet Size, Source/Dest IP Address, Source/Dest Port, Source/ Dest MAC address.

pcapy can do this.

import pcapy
pcap = pcapy.open_offline("httpsession.pcap")

def callback(hdr, data):
    ... do something with hdr and data, which is the captured packet

pcap.loop(0, callback)

Greetings,



From hpj at urpla.net  Wed Dec 28 17:46:41 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Wed, 28 Dec 2016 23:46:41 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <o4191q$vum$1@blaine.gmane.org>
References: <7976395.RisPf5L5KU@xrated> <3560304.JFQfuKfdL7@xrated>
 <o4191q$vum$1@blaine.gmane.org>
Message-ID: <2525992.73WEndpLkm@xrated>

On Mittwoch, 28. Dezember 2016 21:58:38 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> >> Hans-Peter Jansen wrote:
> > It leaves the question on why is Python2 acting as one would expect
> > related to context managers, and Python3 needs this weakref juggling.
> > Maybe something, that's worth to be placed in python-dev. What do you
> > think?
> 
> Well, given
> 
> $ cat mmap_resize.py
> import ctypes
> import mmap
> 
> class T(ctypes.Structure):
>     _fields_ = [("foo", ctypes.c_uint32)]
> 
> SIZE = 2 * mmap.PAGESIZE
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> 
> m = mmap.mmap(f.fileno(), SIZE)
> 
> a = T.from_buffer(m, mmap.PAGESIZE)
> m.resize(mmap.PAGESIZE)
> a.foo = 42
> $ python3 mmap_resize.py
> Traceback (most recent call last):
>   File "mmap_resize.py", line 15, in <module>
>     m.resize(mmap.PAGESIZE)
> BufferError: mmap can't resize with extant buffers exported.
> $ python2.7 mmap_resize.py
> Segmentation fault
> 
> do you really prefer the Python 2 behaviour?

Hmm, so Python2 behavior was "working by chance", or better "working without 
any safety net" which isn't the Real McCoy either in that area.

Now, if only the weakref version would behave with the ctypes objects...

Pete


From diego at vida.com  Wed Dec 28 18:36:05 2016
From: diego at vida.com (Diego Vela)
Date: Wed, 28 Dec 2016 15:36:05 -0800
Subject: Mock object bug with assert_not_called (Steve D'Aprano)
Message-ID: <CAKJZ8r+UnMYSDh184SS14a_RqadqS4cO1SKayi=XE5MGShBM=A@mail.gmail.com>

Re: mock bug.

Python 2.7.8


I'll provide a sketch of the code since it is from work and I'm not allowed
to share it directly.

Sketch of Code:
@patch('emails.emails.render_to_string')
def test_send_email(self, render):
    context = {
          'key': value
    }
    emails.send_email() # calls render_to_string
    render.assert_any_call(
         'template.txt',
         context
     )
    render.assert_not_called()


Expected behavior is that one of the asserts should raise an exception.
The assert_not_called() should assert that the mocked method was not called
at all.  The assert_any_call(vars) asserts there is at least one call with
the given args.  If neither throws an exception then this is a
contradiction because the mocked object, render in this case, has asserted
that there are no calls and that there is at least one call.

Is this sufficient?  I'm not so sure, this is my first time.  Thanks for
the help.

-- 
Diego Vela


From steve+python at pearwood.info  Wed Dec 28 20:14:04 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 29 Dec 2016 12:14:04 +1100
Subject: Mock object bug with assert_not_called (Steve D'Aprano)
References: <CAKJZ8r+UnMYSDh184SS14a_RqadqS4cO1SKayi=XE5MGShBM=A@mail.gmail.com>
 <mailman.21.1482968639.2395.python-list@python.org>
Message-ID: <5864635f$0$1597$c3e8da3$5496439d@news.astraweb.com>

On Thu, 29 Dec 2016 10:36 am, Diego Vela wrote:

> Re: mock bug.
> 
> Python 2.7.8
> 
> 
> I'll provide a sketch of the code since it is from work and I'm not
> allowed to share it directly.

We don't want to see your entire code base. We want to see the smallest,
simplest example that demonstrates the problem. Something we can copy and
paste into a file and run.


> Sketch of Code:
> @patch('emails.emails.render_to_string')
> def test_send_email(self, render):
>     context = {
>           'key': value
>     }
>     emails.send_email() # calls render_to_string
>     render.assert_any_call(
>          'template.txt',
>          context
>      )
>     render.assert_not_called()


This is, I'm afraid, useless. What is "patch"? What is "emails"? What class
does this belong to? What argument "render" should we pass? Where
does "value" come from?

It might help for you to read this:

http://sscce.org/

It is written for Java developers, but the same rules apply here. Your
example code should be 

- as short and simple as you can make it (e.g. since this issue has nothing
to do with sending email, your code should not send email -- a simple print
should be enough for output);

- complete (no missing dependencies, all necessary modules are imported, no
uninitialised variables, functions called with all necessary arguments
specified);

- correct (the code runs and does what you say it does).



Thank you,




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From hpj at urpla.net  Wed Dec 28 22:00:59 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Thu, 29 Dec 2016 04:00:59 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <9101900.Zuuk7oGeDA@xrated>
References: <7976395.RisPf5L5KU@xrated> <3560304.JFQfuKfdL7@xrated>
 <9101900.Zuuk7oGeDA@xrated>
Message-ID: <1659055.2uu0g18TVL@xrated>

On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > > Hans-Peter Jansen wrote:
> > > > Dear Peter,
> > > > 
> > > > thanks for taking valuable time to look into my issue.
> > > 
> > > You're welcome!
> > > 
> > > @contextmanager
> > > def map_struct(m, n):
> > >     m.resize(n * mmap.PAGESIZE)
> > >     keep_me = T.from_buffer(m)
> > >     yield weakref.proxy(keep_me)
> > 
> > Hooray, that did the trick. Great solution, thank you very much!
> 
> Sorry for bothering you again, Peter, but after applying it to the real
> project, that fails consistently similar to:

$ python3 mmap_test_weakref.py 
Traceback (most recent call last):
  File "mmap_test_weakref.py", line 32, in <module>
    assert(bytes(c) == bytes(rest))
AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'


$ cat mmap_test_weakref.py
import ctypes
import mmap
import weakref

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields_ = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n, struct, offset = 0):
    m.resize(n * mmap.PAGESIZE)
    inst = struct.from_buffer(m, offset)
    yield weakref.proxy(inst)

SIZE = mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1, T) as a:
    a.foo = 1
with map_struct(m, 2, T) as b:
    b.foo = 2

offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
    assert(bytes(c) == bytes(rest))


With weakref and mmap.resize() disabled, this acts as expected.
BTW: mmapped files need the first page initialized, the rest is done in the 
kernel (filled with zeros on resize).

Pete


From alec.taylor6 at gmail.com  Thu Dec 29 01:24:26 2016
From: alec.taylor6 at gmail.com (Alec Taylor)
Date: Thu, 29 Dec 2016 17:24:26 +1100
Subject: logger.info / logger.error with logger.addHandler - how to split
 streams?
In-Reply-To: <83439ac0-e4e1-4b37-abd5-e89bb00f5800@googlegroups.com>
References: <CAO+9iGeudrQqF-TnMCPhpk4WKLFq-z41oc5gfU4sjD_cKKnMfg@mail.gmail.com>
 <mailman.263.1482766472.2322.python-list@python.org>
 <83439ac0-e4e1-4b37-abd5-e89bb00f5800@googlegroups.com>
Message-ID: <CAO+9iGeZ+TH3UVPxDogRg-pYh0OPezw8c9vocj+xumBNFzbGLQ@mail.gmail.com>

Thanks

On Tue, Dec 27, 2016 at 2:57 AM, gst <g.starck at gmail.com> wrote:

> Le lundi 26 d?cembre 2016 10:34:48 UTC-5, Alec Taylor a ?crit :
> > So I'm putting .info in one StringIO and .error in another StringIO.
> >
> > How do I stop them from both being put into both?
> >
> > Code: http://ideone.com/Nj6Asz
>
>
> Hi,
>
> it's doable with filter on the handlers:
>
>
> def exact_filter(level):
>     def filter(record):
>         return level == record.levelno
>     filter.filter = filter
>     return filter
>
> stdout_stream.addFilter(exact_filter(logging.INFO))
> stderr_stream.addFilter(exact_filter(logging.ERROR))
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From __peter__ at web.de  Thu Dec 29 03:33:59 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 29 Dec 2016 09:33:59 +0100
Subject: ctypes, memory mapped files and context manager
References: <7976395.RisPf5L5KU@xrated> <3560304.JFQfuKfdL7@xrated>
 <9101900.Zuuk7oGeDA@xrated> <1659055.2uu0g18TVL@xrated>
Message-ID: <o42hpi$70o$1@blaine.gmane.org>

Hans-Peter Jansen wrote:

> On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
>> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
>> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
>> > > Hans-Peter Jansen wrote:
>> > > > Dear Peter,
>> > > > 
>> > > > thanks for taking valuable time to look into my issue.
>> > > 
>> > > You're welcome!
>> > > 
>> > > @contextmanager
>> > > def map_struct(m, n):
>> > >     m.resize(n * mmap.PAGESIZE)
>> > >     keep_me = T.from_buffer(m)
>> > >     yield weakref.proxy(keep_me)
>> > 
>> > Hooray, that did the trick. Great solution, thank you very much!
>> 
>> Sorry for bothering you again, Peter, but after applying it to the real
>> project, that fails consistently similar to:
> 
> $ python3 mmap_test_weakref.py
> Traceback (most recent call last):
>   File "mmap_test_weakref.py", line 32, in <module>
>     assert(bytes(c) == bytes(rest))
> AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'
> 
> 
> $ cat mmap_test_weakref.py
> import ctypes
> import mmap
> import weakref
> 
> from contextlib import contextmanager
> 
> class T(ctypes.Structure):
>     _fields_ = [("foo", ctypes.c_uint32)]
> 
> 
> @contextmanager
> def map_struct(m, n, struct, offset = 0):
>     m.resize(n * mmap.PAGESIZE)
>     inst = struct.from_buffer(m, offset)
>     yield weakref.proxy(inst)
> 
> SIZE = mmap.PAGESIZE
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
> 
> with map_struct(m, 1, T) as a:
>     a.foo = 1
> with map_struct(m, 2, T) as b:
>     b.foo = 2
> 
> offset = ctypes.sizeof(T)
> rest = m.size() - offset
> overhang = ctypes.c_ubyte * rest
> with map_struct(m, 2, overhang, offset) as c:
>     assert(bytes(c) == bytes(rest))
> 
> 
> With weakref and mmap.resize() disabled, this acts as expected.
> BTW: mmapped files need the first page initialized, the rest is done in
> the kernel (filled with zeros on resize).

The minimal example is

>>> import weakref, ctypes
>>> T = ctypes.c_ubyte * 3
>>> t = T()
>>> bytes(t) == b"\0" * 3
True
>>> bytes(weakref.proxy(t)) == b"\0" * 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'

That looks like a leaky abstraction. While I found a workaround

>>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
True

to me your whole approach is beginning to look very questionable. You know, 

"If the implementation is hard to explain, it's a bad idea."

What do you gain from using the mmap/ctypes combo instead of regular file 
operations and the struct module? Your sample code seems to touch every 
single byte of the file once so that there are little to no gains from 
caching. And then your offset is basically a file position managed manually 
instead of implicitly with read, write, and seek.



From andrey.khachaturov at anychart.com  Thu Dec 29 05:13:11 2016
From: andrey.khachaturov at anychart.com (andrey.khachaturov at anychart.com)
Date: Thu, 29 Dec 2016 02:13:11 -0800 (PST)
Subject: Just added AnyChart JS Charts integration templates for easier
 dataviz with Python (+ Flask/Django) and MySQL
Message-ID: <7076a918-cc0a-4ca4-acc1-8ab643f163ea@googlegroups.com>

Hi all,

We at AnyChart JS Charts http://www.anychart.com have just released a series of 20+ integration templates to help web developers add interactive charts, maps, stock and financial graphs, Gantt charts, and dashboards to web apps much easier, no matter what your stack is.

In particular, now there are two templates for Python in our Technical Integration collection http://www.anychart.com/integrations/, all distributed under the Apache 2.0 License and forkable on GitHub:

1) Python, Flask and MySQL
https://github.com/anychart-integrations/python-flask-mysql-template

2) Python, Django and MySQL
https://github.com/anychart-integrations/python-django-mysql-template

You are welcome to check those out and ask your questions if any.

Thanks.


From hpj at urpla.net  Thu Dec 29 07:18:09 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Thu, 29 Dec 2016 13:18:09 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <o42hpi$70o$1@blaine.gmane.org>
References: <7976395.RisPf5L5KU@xrated> <1659055.2uu0g18TVL@xrated>
 <o42hpi$70o$1@blaine.gmane.org>
Message-ID: <8107844.xiv98UW103@xrated>

On Donnerstag, 29. Dezember 2016 09:33:59 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
> 
> The minimal example is
> 
> >>> import weakref, ctypes
> >>> T = ctypes.c_ubyte * 3
> >>> t = T()
> >>> bytes(t) == b"\0" * 3
> 
> True
> 
> >>> bytes(weakref.proxy(t)) == b"\0" * 3
> 
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
> 
> That looks like a leaky abstraction. While I found a workaround
> 
> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
> 
> True

I found a couple of other rough corners already, when working with the ctypes 
module. Obviously, this module is lacking some love.

> to me your whole approach is beginning to look very questionable. You know,
> 
> "If the implementation is hard to explain, it's a bad idea."
> 
> What do you gain from using the mmap/ctypes combo instead of regular file
> operations and the struct module? Your sample code seems to touch every
> single byte of the file once so that there are little to no gains from
> caching. And then your offset is basically a file position managed manually
> instead of implicitly with read, write, and seek.

Of course, the real code is a bit more complex... The code presented here is 
for demonstration purposes only. I'm not allowed to reveal the projects' code, 
but I can state, that using this combination allows for crawling through huge 
files (5-25GB) in unbelievable performance (without any further optimization), 
and updating parts of it. By delegating the whole I/O management to the 
kernel, one can observe, that python runs at full speed managing the data just 
by reference and assignment operations, all (mostly) in place. The resource 
usage is impressively low at the same time. Since the code is meant to be 
executed with many instances in parallel on a single machine, this is an 
important design criteria. 

While I would love to get rid of these dreaded and unpythonic del statements, 
I can accept them for now, until a better approach is found. 

Will dig through the ctypes module again, when I find time.

Thanks again for taking your valuable time, Peter. Much appreciated.

I wish you a Happy New Year!

Cheers,
Pete


From valkrem at yahoo.com  Thu Dec 29 10:46:13 2016
From: valkrem at yahoo.com (Val Krem)
Date: Thu, 29 Dec 2016 15:46:13 +0000 (UTC)
Subject: data
References: <478662942.3034497.1483026373587.ref@mail.yahoo.com>
Message-ID: <478662942.3034497.1483026373587@mail.yahoo.com>

Hi all,
I have a sample of data set and would  like to summarize in the following way.


ID,class,y
1,12,10
1,12,10
1,12,20
1,13,20
1,13,10
1,13,10
1,14,20
2,21,20
2,21,20
2,21,10
2,23,10
2,23,20
2,34,20
2,34,10
2,35,10

I want  get  the total count by ID, and the  the number of classes
by ID. The y variable is either 10 or 20 and count each by iD

The result should look like as follows.

ID,class,count,10's,20's
1,3,7,4,3
2,4,8,4,4

I can do this in  two or more steps. Is there an efficient way of doing it?


I used 

pd.crosstab(a['ID'],a['y'],margins=True)
and got

ID,10's,20's all
1,4,3,7
2,4,4,8

but I want get the class count as well like as follows

ID,class,10's,20's,all
1,3,4,3,7
2,4,4,4,8

how do I do it in python?
thank you in advance


From larry.martell at gmail.com  Thu Dec 29 15:45:17 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Thu, 29 Dec 2016 15:45:17 -0500
Subject: sorting strings numerically while dealing with missing values
In-Reply-To: <CALwzid==a0NPxSzik2zA-McKfkgHFNtA6wNPNssGskc4rMmnvg@mail.gmail.com>
References: <CACwCsY5fj4pPYfD8enhRXFp=s9_OH0tD9KArhnS-HQzJ=gnPJw@mail.gmail.com>
 <CALwzid==a0NPxSzik2zA-McKfkgHFNtA6wNPNssGskc4rMmnvg@mail.gmail.com>
Message-ID: <CACwCsY6rihJ5BdVpP2Vzg359JnOiegm9SfegRgp3C69_dztw3Q@mail.gmail.com>

On Wed, Dec 28, 2016 at 3:43 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> On Wed, Dec 28, 2016 at 2:14 PM, Larry Martell <larry.martell at gmail.com> wrote:
>>
>> I have a list containing a list of strings that I want to sort
>> numerically by one of the fields. I am doing this:
>>
>> sorted(rows, key=float(itemgetter(sortby)))
>
> I'm guessing that you left out a lambda here since the key argument
> takes a function.
>
>> Which works fine as long as all the sort keys convert to a float.
>> Problem is that some are blank or None and those throw an exception.
>> How can I handle that case and still sort? I'd want the blank or None
>> fields to come out either at the beginning or end of the sorted list
>> (not sure what the customer wants for this yet).
>
>
> def sort_key(sortby, none_first=False):
>     def key(row):
>         try:
>             value = float(row[sortby])
>         except ValueError:
>             value = None
>         return ((value is None) != none_first, value)
>     return key
>
> sorted(rows, key=sort_key(4, none_first=True))

Thanks. I got this working using a function similar to this.


From eryksun at gmail.com  Thu Dec 29 16:27:56 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 29 Dec 2016 21:27:56 +0000
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <8107844.xiv98UW103@xrated>
References: <7976395.RisPf5L5KU@xrated> <1659055.2uu0g18TVL@xrated>
 <o42hpi$70o$1@blaine.gmane.org> <8107844.xiv98UW103@xrated>
Message-ID: <CACL+1asy-WBMuW8VfgL6NQLTxcOphqnu6z8iu1hnJiNkN=8Dcg@mail.gmail.com>

On Thu, Dec 29, 2016 at 12:18 PM, Hans-Peter Jansen <hpj at urpla.net> wrote:
>> >>> import weakref, ctypes
>> >>> T = ctypes.c_ubyte * 3
>> >>> t = T()
>> >>> bytes(t) == b"\0" * 3
>>
>> True
>>
>> >>> bytes(weakref.proxy(t)) == b"\0" * 3
>>
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
>>
>> That looks like a leaky abstraction. While I found a workaround
>>
>> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
>>
>> True
>
> I found a couple of other rough corners already, when working with the ctypes
> module. Obviously, this module is lacking some love.

That's not the fault of ctypes. There's no requirement for objects
that implement the buffer protocol to also implement __bytes__. You'd
have the same problem if you tried to proxy a memoryview.

However, using a proxy seems particularly worthless for ctypes. Type
checking is integral to the design of ctypes, and a weakproxy won't
work:

    >>> a = (ctypes.c_ubyte * 3)()
    >>> ctypes.addressof(a)
    139959173036992
    >>> ctypes.pointer(a)
    <__main__.LP_c_ubyte_Array_3 object at 0x7f4ac8caba60>
    >>> ctypes.string_at(a, 3)
    b'\x00\x00\x00'

    >>> p = weakref.proxy(a)
    >>> type(p)
    <class 'weakproxy'>

    >>> ctypes.addressof(p)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: invalid type

    >>> ctypes.pointer(p)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: _type_ must have storage info

    >>> ctypes.string_at(p, 3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3.5/ctypes/__init__.py", line 491, in string_at
        return _string_at(ptr, size)
    ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type


From subhabangalore at gmail.com  Thu Dec 29 17:05:46 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Thu, 29 Dec 2016 14:05:46 -0800 (PST)
Subject: UTF-8 Encoding Error
In-Reply-To: <mailman.260.1482703640.2322.python-list@python.org>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
 <20161225055016.GI16063@creation>
 <CAF40P_3c48POB24cmdhR6e5gTCPjq1i7ROMKfwKyZHGT48Smtw@mail.gmail.com>
 <mailman.260.1482703640.2322.python-list@python.org>
Message-ID: <610a0e37-f25e-404b-a790-aba057444392@googlegroups.com>

On Monday, December 26, 2016 at 3:37:37 AM UTC+5:30, Gonzalo V wrote:
> Try utf-8-sig
> El 25 dic. 2016 2:57 AM, "Grady Martin" <> escribi?:
> 
> > On 2016?12?22? 22?38?,  wrote:
> >
> >> I am getting the error:
> >> UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15:
> >> invalid start byte
> >>
> >
> > The following is a reflex of mine, whenever I encounter Python 2 Unicode
> > errors:
> >
> > import sys
> > reload(sys)
> > sys.setdefaultencoding('utf8')
> >
> > A relevant Stack Exchange thread awaits you here:
> >
> > http://stackoverflow.com/a/21190382/2230956
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >

Thank you for your kind time and answers. 

I tried to open one file in default ASCII format in MS-Windows 7. 
txtf=open("/python27/TestFiles/small_file_1.txt","r").read()
I could write them in UTF-8 using 
cd1=codecs.open("/python27/TestFiles/file1.pos","w", "utf-8-sig")
cd1.write(txtf)

Here, I was getting an error as,
UnicodeDecodeError: 'ascii' codec can't decode byte 0x96 in position 150: ordinal not in range(128)

Then I used,
>>> import sys
>>> reload(sys)
>>> sys.setdefaultencoding('utf8')

and then wrote 
>>> cd1.write(txtf)
it went fine. 

Now in my actual problem I am writing it bit differently: 

with open('textfile.txt') as f:
        for i, g in enumerate(grouper(n, f, fillvalue=''), 1):
            with open('/Python27/TestFiles/small_filing_{0}.pos'.format(i * n), 'w') as fout:
                fout.writelines(g)

I am trying to fix this. 

If you may kindly suggest. 




From subhabangalore at gmail.com  Thu Dec 29 17:19:57 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Thu, 29 Dec 2016 14:19:57 -0800 (PST)
Subject: UTF-8 Encoding Error
In-Reply-To: <610a0e37-f25e-404b-a790-aba057444392@googlegroups.com>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
 <20161225055016.GI16063@creation>
 <CAF40P_3c48POB24cmdhR6e5gTCPjq1i7ROMKfwKyZHGT48Smtw@mail.gmail.com>
 <mailman.260.1482703640.2322.python-list@python.org>
 <610a0e37-f25e-404b-a790-aba057444392@googlegroups.com>
Message-ID: <e99bb31f-a8c0-4747-b560-ab8780e6e265@googlegroups.com>

On Friday, December 30, 2016 at 3:35:56 AM UTC+5:30, subhaba... at gmail.com wrote:
> On Monday, December 26, 2016 at 3:37:37 AM UTC+5:30, Gonzalo V wrote:
> > Try utf-8-sig
> > El 25 dic. 2016 2:57 AM, "Grady Martin" <> escribi?:
> > 
> > > On 2016?12?22? 22?38?,  wrote:
> > >
> > >> I am getting the error:
> > >> UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15:
> > >> invalid start byte
> > >>
> > >
> > > The following is a reflex of mine, whenever I encounter Python 2 Unicode
> > > errors:
> > >
> > > import sys
> > > reload(sys)
> > > sys.setdefaultencoding('utf8')
> > >
> > > A relevant Stack Exchange thread awaits you here:
> > >
> > > http://stackoverflow.com/a/21190382/2230956
> > > --
> > > https://mail.python.org/mailman/listinfo/python-list
> > >
> 
> Thank you for your kind time and answers. 
> 
> I tried to open one file in default ASCII format in MS-Windows 7. 
> txtf=open("/python27/TestFiles/small_file_1.txt","r").read()
> I could write them in UTF-8 using 
> cd1=codecs.open("/python27/TestFiles/file1.pos","w", "utf-8-sig")
> cd1.write(txtf)
> 
> Here, I was getting an error as,
> UnicodeDecodeError: 'ascii' codec can't decode byte 0x96 in position 150: ordinal not in range(128)
> 
> Then I used,
> >>> import sys
> >>> reload(sys)
> >>> sys.setdefaultencoding('utf8')
> 
> and then wrote 
> >>> cd1.write(txtf)
> it went fine. 
> 
> Now in my actual problem I am writing it bit differently: 
> 
> with open('textfile.txt') as f:
>         for i, g in enumerate(grouper(n, f, fillvalue=''), 1):
>             with open('/Python27/TestFiles/small_filing_{0}.pos'.format(i * n), 'w') as fout:
>                 fout.writelines(g)
> 
> I am trying to fix this. 
> 
> If you may kindly suggest.

The grouper method is:
def grouper(n, iterable, fillvalue=None):
        "Collect data into fixed-length chunks or blocks"
        args = [iter(iterable)] * n
        return izip_longest(fillvalue=fillvalue, *args)
n = 3


From hpj at urpla.net  Thu Dec 29 19:28:12 2016
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Fri, 30 Dec 2016 01:28:12 +0100
Subject: ctypes, memory mapped files and context manager
In-Reply-To: <CACL+1asy-WBMuW8VfgL6NQLTxcOphqnu6z8iu1hnJiNkN=8Dcg@mail.gmail.com>
References: <7976395.RisPf5L5KU@xrated> <8107844.xiv98UW103@xrated>
 <CACL+1asy-WBMuW8VfgL6NQLTxcOphqnu6z8iu1hnJiNkN=8Dcg@mail.gmail.com>
Message-ID: <3452914.eWgquO5k8a@xrated>

Dear Eryk,

thanks for chiming in.

On Donnerstag, 29. Dezember 2016 21:27:56 eryk sun wrote:
> On Thu, Dec 29, 2016 at 12:18 PM, Hans-Peter Jansen <hpj at urpla.net> wrote:
> >> >>> import weakref, ctypes
> >> >>> T = ctypes.c_ubyte * 3
> >> >>> t = T()
> >> >>> bytes(t) == b"\0" * 3
> >> 
> >> True
> >> 
> >> >>> bytes(weakref.proxy(t)) == b"\0" * 3
> >> 
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> 
> >> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
> >> 
> >> That looks like a leaky abstraction. While I found a workaround
> >> 
> >> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
> >> 
> >> True
> > 
> > I found a couple of other rough corners already, when working with the
> > ctypes module. Obviously, this module is lacking some love.
> 
> That's not the fault of ctypes. There's no requirement for objects
> that implement the buffer protocol to also implement __bytes__. You'd
> have the same problem if you tried to proxy a memoryview.
> 
> However, using a proxy seems particularly wHaorthless for ctypes. Type
> checking is integral to the design of ctypes, and a weakproxy won't

Did you follow the discussion? 

I'm trying to make context manager work with ctypes.from_buffer on mmapped 
files:

import ctypes
import mmap
import weakref

NOPROB=False
#NOPROB=True

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields_ = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n, struct, offset = 0):
    m.resize(n * mmap.PAGESIZE)
    inst = struct.from_buffer(m, offset)
    yield inst

SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1, T) as a:
    a.foo = 1
if NOPROB:
    del a
with map_struct(m, 2, T) as b:
    b.foo = 2
if NOPROB:
    del b

offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
    assert(bytes(c) == bytes(rest))
if NOPROB:
    del c
 

Without these dreaded del statements, this code doesn't work:

$ python3 mmap_test2.py 
Traceback (most recent call last):
  File "mmap_test2.py", line 30, in <module>
    with map_struct(m, 2, T) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test2.py", line 16, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.

It will work, if you define NOPROB=True.

The weakref approach was an attempt to make this work.

Do you have an idea how to create the context manager in a way, that obsoletes 
these ugly dels? Something like ctypes.from_buffer_release() that is able to 
actively release the mapping is needed here, AFAICS.

This code works with Python2 due to the mmap module not checking for any 
existing mappings which may lead to segfaults, if the mmap is resized.

Thanks,
Pete



From steve+python at pearwood.info  Thu Dec 29 20:46:03 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 30 Dec 2016 12:46:03 +1100
Subject: UTF-8 Encoding Error
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
 <20161225055016.GI16063@creation>
 <mailman.258.1482645261.2322.python-list@python.org>
Message-ID: <5865bc5e$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Sun, 25 Dec 2016 04:50 pm, Grady Martin wrote:

> On 2016?12?22? 22?38?, subhabangalore at gmail.com wrote:
>>I am getting the error:
>>UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15:
>>invalid start byte
> 
> The following is a reflex of mine, whenever I encounter Python 2 Unicode
> errors:
> 
> import sys
> reload(sys)
> sys.setdefaultencoding('utf8')


This is a BAD idea, and doing it by "reflex" without very careful thought is
just cargo-cult programming. You should not thoughtlessly change the
default encoding without knowing what you are doing -- and if you know what
you are doing, you won't change it at all.

The Python interpreter *intentionally* removes setdefaultencoding at startup
for a reason. Changing the default encoding can break the interpreter, and
it is NEVER what you actually need. If you think you want it because it
fixes "Unicode errors", all you are doing is covering up bugs in your code.

Here is some background on why setdefaultencoding exists, and why it is
dangerous:

https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/

If you have set the Python 2 default encoding to anything but ASCII, you are
now running a broken system with subtle bugs, including in data structures
as fundamental as dicts.

The standard behaviour:

py> d = {u'caf?': 1}
py> for key in d:
...     print key == 'caf\xc3\xa9'
...
False


As we should expect: the key in the dict, u'caf?', is *not* the same as the
byte-string 'caf\xc3\xa9'. But watch how we can break dictionaries by
changing the default encoding:

py> reload(sys)
<module 'sys' (built-in)>
py> sys.setdefaultencoding('utf-8')  # don't do this
py> for key in d:
...     print key == 'caf\xc3\xa9'
...
True


So Python now thinks that 'caf\xc3\xa9' is a key. Or does it?

py> d['caf\xc3\xa9']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'caf\xc3\xa9'

By changing the default encoding, we now have something which is both a key
and not a key of the dict at the same time.



> A relevant Stack Exchange thread awaits you here:
> 
> http://stackoverflow.com/a/21190382/2230956

And that's why I don't trust StackOverflow. It's not bad for answering
simple questions, but once the question becomes more complex the quality of
accepted answers goes down the toilet. The highest voted answer is *wrong*
and *dangerous*.

And then there's this comment:

    Until this moment I was forced to include "# -- coding: utf-8 --" at 
    the begining of each document. This is way much easier and works as
    charm

I have no words for how wrong that is. And this comment:

    ty, this worked for my problem with python throwing UnicodeDecodeError
    on var = u"""vary large string"""

No it did not. There is no possible way that Python will throw that
exception on assignment to a Unicode string literal.

It is posts like this that demonstrate how untrustworthy StackOverflow can
be.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From subhabangalore at gmail.com  Fri Dec 30 01:11:56 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Thu, 29 Dec 2016 22:11:56 -0800 (PST)
Subject: UTF-8 Encoding Error
In-Reply-To: <5865bc5e$0$1596$c3e8da3$5496439d@news.astraweb.com>
References: <98623c65-e2fc-4c39-a8ac-b758fe5318a3@googlegroups.com>
 <20161225055016.GI16063@creation>
 <mailman.258.1482645261.2322.python-list@python.org>
 <5865bc5e$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <7c38d162-66b8-49de-99af-b2da640628ab@googlegroups.com>

On Friday, December 30, 2016 at 7:16:25 AM UTC+5:30, Steve D'Aprano wrote:
> On Sun, 25 Dec 2016 04:50 pm, Grady Martin wrote:
> 
> > On 2016?12?22? 22?38?, wrote:
> >>I am getting the error:
> >>UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15:
> >>invalid start byte
> > 
> > The following is a reflex of mine, whenever I encounter Python 2 Unicode
> > errors:
> > 
> > import sys
> > reload(sys)
> > sys.setdefaultencoding('utf8')
> 
> 
> This is a BAD idea, and doing it by "reflex" without very careful thought is
> just cargo-cult programming. You should not thoughtlessly change the
> default encoding without knowing what you are doing -- and if you know what
> you are doing, you won't change it at all.
> 
> The Python interpreter *intentionally* removes setdefaultencoding at startup
> for a reason. Changing the default encoding can break the interpreter, and
> it is NEVER what you actually need. If you think you want it because it
> fixes "Unicode errors", all you are doing is covering up bugs in your code.
> 
> Here is some background on why setdefaultencoding exists, and why it is
> dangerous:
> 
> https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/
> 
> If you have set the Python 2 default encoding to anything but ASCII, you are
> now running a broken system with subtle bugs, including in data structures
> as fundamental as dicts.
> 
> The standard behaviour:
> 
> py> d = {u'caf?': 1}
> py> for key in d:
> ...     print key == 'caf\xc3\xa9'
> ...
> False
> 
> 
> As we should expect: the key in the dict, u'caf?', is *not* the same as the
> byte-string 'caf\xc3\xa9'. But watch how we can break dictionaries by
> changing the default encoding:
> 
> py> reload(sys)
> <module 'sys' (built-in)>
> py> sys.setdefaultencoding('utf-8')  # don't do this
> py> for key in d:
> ...     print key == 'caf\xc3\xa9'
> ...
> True
> 
> 
> So Python now thinks that 'caf\xc3\xa9' is a key. Or does it?
> 
> py> d['caf\xc3\xa9']
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> KeyError: 'caf\xc3\xa9'
> 
> By changing the default encoding, we now have something which is both a key
> and not a key of the dict at the same time.
> 
> 
> 
> > A relevant Stack Exchange thread awaits you here:
> > 
> > http://stackoverflow.com/a/21190382/2230956
> 
> And that's why I don't trust StackOverflow. It's not bad for answering
> simple questions, but once the question becomes more complex the quality of
> accepted answers goes down the toilet. The highest voted answer is *wrong*
> and *dangerous*.
> 
> And then there's this comment:
> 
>     Until this moment I was forced to include "# -- coding: utf-8 --" at 
>     the begining of each document. This is way much easier and works as
>     charm
> 
> I have no words for how wrong that is. And this comment:
> 
>     ty, this worked for my problem with python throwing UnicodeDecodeError
>     on var = u"""vary large string"""
> 
> No it did not. There is no possible way that Python will throw that
> exception on assignment to a Unicode string literal.
> 
> It is posts like this that demonstrate how untrustworthy StackOverflow can
> be.
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.

Thanks for your detailed comment. The code is going all fine sometimes, and sometimes giving out errors. If any one may see how I am doing the problem.


From abaco121 at gmail.com  Fri Dec 30 02:19:36 2016
From: abaco121 at gmail.com (abaco121 at gmail.com)
Date: Thu, 29 Dec 2016 23:19:36 -0800 (PST)
Subject: Obtain javascript result
In-Reply-To: <5b51f0f9-bc69-4b9a-bd55-9fa0a9117341@googlegroups.com>
References: <7b27e2ef-4abd-4d90-86b9-2c1c428b74eb@googlegroups.com>
 <5b51f0f9-bc69-4b9a-bd55-9fa0a9117341@googlegroups.com>
Message-ID: <9f60b5d9-415d-4381-ba72-2c788a321d93@googlegroups.com>

Hi,
there's a problem in betexplorer?
this php page dont response anything to get odds

http://www.betexplorer.com/soccer/russia/youth-\league/matchdetails.php?matchid=rLu2Xsdi

from 24 december dont work

thanxs



Il giorno domenica 23 ottobre 2016 20:09:30 UTC+2, epr... at gmail.com ha scritto:
> Ok, I solved to this way:
> 
> from bs4 import BeautifulSoup
> from selenium import webdriver
> 
> driver = webdriver.Chrome()
> driver.get('http://www.betexplorer.com/soccer/russia/youth-\league/matchdetails.php?matchid=rLu2Xsdi')
> 
> pg_src = driver.page_source
> driver.close()
> soup = BeautifulSoup(pg_src, 'html.parser')
> # start from here I do something with soup ...
> 
> Windows 10 / Python 3.5.2
> 
> Thanks


From redstone-cold at 163.com  Fri Dec 30 02:47:48 2016
From: redstone-cold at 163.com (iMath)
Date: Thu, 29 Dec 2016 23:47:48 -0800 (PST)
Subject: python3 - set '\n\n' as the terminator when writing a formatted
 LogRecord
Message-ID: <3d773bbf-f1d9-421d-963f-1b03671bfdcc@googlegroups.com>

Is it possible to set '\n\n' as the terminator when writing a formatted LogRecord to a stream by changing the format parameter of logging.basicConfig?

I know it is possible using the terminator attribute of StreamHandler class to implement this, I just wonder Is it possible to achieve this feature by changing the format parameter? I am not familiar with the format string language


From __peter__ at web.de  Fri Dec 30 04:55:39 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 30 Dec 2016 10:55:39 +0100
Subject: python3 - set '\n\n' as the terminator when writing a formatted
 LogRecord
References: <3d773bbf-f1d9-421d-963f-1b03671bfdcc@googlegroups.com>
Message-ID: <o45aum$6bc$1@blaine.gmane.org>

iMath wrote:

> Is it possible to set '\n\n' as the terminator when writing a formatted
> LogRecord to a stream by changing the format parameter of
> logging.basicConfig?
> 
> I know it is possible using the terminator attribute of StreamHandler
> class to implement this, I just wonder Is it possible to achieve this
> feature by changing the format parameter? I am not familiar with the
> format string language

How about altering an existing format?

>>> logging.basicConfig(format=logging.BASIC_FORMAT + "\n", 
level=logging.INFO)
>>> g = logging.getLogger()
>>> g.info("hi")
INFO:root:hi

>>> 




From __peter__ at web.de  Fri Dec 30 05:08:19 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 30 Dec 2016 11:08:19 +0100
Subject: ctypes, memory mapped files and context manager
References: <7976395.RisPf5L5KU@xrated> <1659055.2uu0g18TVL@xrated>
 <o42hpi$70o$1@blaine.gmane.org> <8107844.xiv98UW103@xrated>
 <CACL+1asy-WBMuW8VfgL6NQLTxcOphqnu6z8iu1hnJiNkN=8Dcg@mail.gmail.com>
Message-ID: <o45bmh$f44$1@blaine.gmane.org>

eryk sun wrote:

> On Thu, Dec 29, 2016 at 12:18 PM, Hans-Peter Jansen <hpj at urpla.net> wrote:
>>> >>> import weakref, ctypes
>>> >>> T = ctypes.c_ubyte * 3
>>> >>> t = T()
>>> >>> bytes(t) == b"\0" * 3
>>>
>>> True
>>>
>>> >>> bytes(weakref.proxy(t)) == b"\0" * 3
>>>
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
>>>
>>> That looks like a leaky abstraction. While I found a workaround
>>>
>>> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
>>>
>>> True
>>
>> I found a couple of other rough corners already, when working with the
>> ctypes module. Obviously, this module is lacking some love.
> 
> That's not the fault of ctypes. 

By adding a Structure.close() method that would release the underlying 
memoryview reference and make all further attribute access fail ctypes could 
support Hans-Peter's use case quite easily. In that sense I think it is the 
"fault" or rather a limitation of ctypes. I'm not sure though whether ctypes 
is the right tool here in the first place. 

Anyway, as long as he gets enough advantages from his current approach a few 
odd del statements might be OK.



From as at sci.fi  Fri Dec 30 06:47:06 2016
From: as at sci.fi (Anssi Saari)
Date: Fri, 30 Dec 2016 13:47:06 +0200
Subject: [OT] Security question
References: <o3g700$vm2$1@blaine.gmane.org>
Message-ID: <vg337h58wqt.fsf@coffee.modeemi.fi>

"Frank Millman" <frank at chagford.com> writes:

> Hi all
>
> This is off-topic, but I would appreciate a comment on this matter.
>
> I have just upgraded my internet connection from ADSL to Fibre.
>
> As part of the process, my ISP sent a text message to my cell phone
> with the username and password I must use to connect.
>
> To my surprise, they sent me my existing username *and* my existing
> password, all in clear text.

I'd say it depends on what the password is actually used for. You seem
to indicate it's just so you can access the internet? To me it seems
abusing that password is hard to impossible since it's your fibre to
your home. If the password is used for access control for anything then
it's an awful practise.

In my case, I have one password for the email account my ISP provides
and another for their web management pages where I can buy more or get
rid of services and see my bills and such.



From marko at pacujo.net  Fri Dec 30 06:56:46 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 30 Dec 2016 13:56:46 +0200
Subject: [OT] Security question
References: <o3g700$vm2$1@blaine.gmane.org> <vg337h58wqt.fsf@coffee.modeemi.fi>
 <mailman.32.1483098442.2395.python-list@python.org>
Message-ID: <871swp1vgh.fsf@elektro.pacujo.net>

Anssi Saari <as at sci.fi>:

> "Frank Millman" <frank at chagford.com> writes:
>> To my surprise, they sent me my existing username *and* my existing
>> password, all in clear text.
>
> I'd say it depends on what the password is actually used for. You seem
> to indicate it's just so you can access the internet? To me it seems
> abusing that password is hard to impossible since it's your fibre to
> your home. If the password is used for access control for anything
> then it's an awful practise.

The message to take home is that whenever you are faced with a password
prompt, the recipient can do with the password whatever they want. You
should assume the worst. The password will be stored in the clear and
all employees of the recipient have free access to it. Also, there's a
high likelihood that the credentials will leak outside the organization.


Marko


From rosuav at gmail.com  Fri Dec 30 07:26:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 30 Dec 2016 23:26:28 +1100
Subject: [OT] Security question
In-Reply-To: <vg337h58wqt.fsf@coffee.modeemi.fi>
References: <o3g700$vm2$1@blaine.gmane.org> <vg337h58wqt.fsf@coffee.modeemi.fi>
Message-ID: <CAPTjJmqM5fdOwDqh9REqj4wLNK4Ne=Egi6qARKnEgpo_Oie1xQ@mail.gmail.com>

On Fri, Dec 30, 2016 at 10:47 PM, Anssi Saari <as at sci.fi> wrote:
> I'd say it depends on what the password is actually used for. You seem
> to indicate it's just so you can access the internet? To me it seems
> abusing that password is hard to impossible since it's your fibre to
> your home. If the password is used for access control for anything then
> it's an awful practise.

"Just" so he can access the internet? That's no small deal. If someone
else can sign in with the same password, s/he can do any sort of abuse
and it'll be registered to someone else. What spammer wouldn't jump at
the chance to blame someone else for the traffic?

ChrisA


From steve+python at pearwood.info  Fri Dec 30 09:47:32 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 01:47:32 +1100
Subject: Simulating int arithmetic with wrap-around
Message-ID: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>

It's easy to simulate integer arithmetic with wrap-around for *unsigned*
ints. For example, imagine a four-bit integer (0 through 15):

# Addition
py> (7 + 8) & 0xF
15
py> (7 + 9) & 0xF
0
py> (7 + 10) & 0xF
1

# Multiplication
py> (7 * 2) & 0xF
14
py> (7 * 3) & 0xF
5
py> (7 * 4) & 0xF
12


And in general, for any operator ?, and a and b both in the range 0 through
2**N-1, we can simulate unsigned N-bit integer arithmetic with:

    a?b & (2**N - 1)

(I think this works for all the arithmetic operators, and the bitwise
operations. The bitmask & 2**N-1 is not needed for the bitwise operations
except for left-shift.)


How about *signed* integers?

7 + 1 => -8
7 + 2 => -7
7 + 7 => -2

7 * 2 => -2
7 * 3 => 5
7 * 4 => -4


Again, assume both operands are in range for an N-bit signed integer. What's
a good way to efficiently, or at least not too inefficiently, do the
calculations in Python?


Signed arithmetic also has some gotchas. For example, -x is not necessarily
defined, nor is abs(x).


Thanks,





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From marko at pacujo.net  Fri Dec 30 10:03:02 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 30 Dec 2016 17:03:02 +0200
Subject: [OT] Security question
References: <o3g700$vm2$1@blaine.gmane.org> <vg337h58wqt.fsf@coffee.modeemi.fi>
 <CAPTjJmqM5fdOwDqh9REqj4wLNK4Ne=Egi6qARKnEgpo_Oie1xQ@mail.gmail.com>
 <mailman.33.1483100791.2395.python-list@python.org>
Message-ID: <87wpehzcgp.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Fri, Dec 30, 2016 at 10:47 PM, Anssi Saari <as at sci.fi> wrote:
>> I'd say it depends on what the password is actually used for. You seem
>> to indicate it's just so you can access the internet? To me it seems
>> abusing that password is hard to impossible since it's your fibre to
>> your home. If the password is used for access control for anything then
>> it's an awful practise.
>
> "Just" so he can access the internet? That's no small deal. If someone
> else can sign in with the same password, s/he can do any sort of abuse
> and it'll be registered to someone else. What spammer wouldn't jump at
> the chance to blame someone else for the traffic?

That's called plausible deniability.


Marko


From storchaka at gmail.com  Fri Dec 30 10:14:00 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 30 Dec 2016 17:14:00 +0200
Subject: Simulating int arithmetic with wrap-around
In-Reply-To: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o45tji$fc7$1@blaine.gmane.org>

On 30.12.16 16:47, Steve D'Aprano wrote:
> Again, assume both operands are in range for an N-bit signed integer. What's
> a good way to efficiently, or at least not too inefficiently, do the
> calculations in Python?

def to_unsigned(bits, x):
     return x & ((1<<bits)-1)

def to_signed(bits, x):
     offset = 1<<(bits-1)
     return to_unsigned(bits, x + offset) - offset




From rosuav at gmail.com  Fri Dec 30 10:27:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 31 Dec 2016 02:27:56 +1100
Subject: Simulating int arithmetic with wrap-around
In-Reply-To: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpGZPwc2N35jdSR68s-Jt5ijX36hFTBCTKM9wTPQ0Xw3g@mail.gmail.com>

On Sat, Dec 31, 2016 at 1:47 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> How about *signed* integers?
>
> 7 + 1 => -8
> 7 + 2 => -7
> 7 + 7 => -2
>
> 7 * 2 => -2
> 7 * 3 => 5
> 7 * 4 => -4
>
>
> Again, assume both operands are in range for an N-bit signed integer. What's
> a good way to efficiently, or at least not too inefficiently, do the
> calculations in Python?

One way would be to work with the integers as unsigned numbers, and
then render them signed for display only. The neat thing about two's
complement (as opposed to two's compliment, which is a really sweet
thing to say to someone) is that a lot of operations work exactly the
same way. You add 7+2 and get 9, and then display 9 as -7. You take
that 9 (really -7) and add 5 to it, and you get 14, which you display
as -2. Add another 4 and you get 18, mask that off and you get 2. You
may have to handle multiplication and division differently, I think,
and you might need a sign-extend right shift operator (as opposed to a
zero-fill right shift), but they should be able to be defined in terms
of the other operations.

7 * 2 => 14, displayed as -2
7 * 3 => 21, mask to 5
7 * 4 => 28, mask to 12, display as -4
7 * 9 => 63, mask to 15, display as -1. Conceptually 7*-7 => -49.

Actually you might be able to get away with multiplication perfectly.
I'm thinking of the Intel 80x86 opcodes, where MUL and IMUL are
unsigned and signed multiplication, respectively, but they differ
because the result is twice as wide as the operands (eg if you
multiply two 16-bit numbers, you get a 32-bit result). The Intel chips
do the same with division (eg you divide a 32-bit number by a 16-bit
and get back a 16-bit quotient and 16-bit remainder). You may be able
to take the exact same short-cut.

> Signed arithmetic also has some gotchas. For example, -x is not necessarily
> defined, nor is abs(x).

AIUI -x is always defined, but might be equal to x in two
circumstances (zero and MAXINT+1). Not sure about abs(x), but probably
would be the same.

ChrisA


From grant.b.edwards at gmail.com  Fri Dec 30 12:00:02 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 30 Dec 2016 17:00:02 +0000 (UTC)
Subject: Parse a Wireshark pcap file
References: <mailman.853.1358904722.2939.python-list@python.org>
 <236debf2-d81d-4d54-83a7-7cbb50184853@googlegroups.com>
Message-ID: <o463qi$h8j$1@blaine.gmane.org>

On 2016-12-27, 1991manish.kumar at gmail.com <1991manish.kumar at gmail.com> wrote:
>
> I have a pcap file, I want to parse that file & fetch some
> information like Timestamp, Packet Size, Source/Dest IP Address,
> Source/Dest Port, Source/ Dest MAC address.

I've been using pylibpcap for ages.  It's a bit old, but still works
fine for me:

 https://sourceforge.net/projects/pylibpcap/

There's also pypcap:

 https://github.com/pynetwork/pypcap

-- 
Grant Edwards               grant.b.edwards        Yow! Now KEN and BARBIE
                                  at               are PERMANENTLY ADDICTED to
                              gmail.com            MIND-ALTERING DRUGS ...



From einstein1410 at gmail.com  Fri Dec 30 12:50:19 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 09:50:19 -0800 (PST)
Subject: learning and experimenting python.
Message-ID: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>

Hello everyone,
I am the new comer and learner of python.
I have a doubt that when I type python and press enter it shows a prompt like
>>>
But why it is >>> ?
Is there any special reason?
Why it is not setted as @,& or any other special characters?


From ian.g.kelly at gmail.com  Fri Dec 30 13:44:17 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 30 Dec 2016 12:44:17 -0600
Subject: learning and experimenting python.
In-Reply-To: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
Message-ID: <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>

On Fri, Dec 30, 2016 at 11:50 AM,  <einstein1410 at gmail.com> wrote:
> Hello everyone,
> I am the new comer and learner of python.
> I have a doubt that when I type python and press enter it shows a prompt like
>>>>
> But why it is >>> ?
> Is there any special reason?
> Why it is not setted as @,& or any other special characters?

Because it's easy to recognize as a prompt, I would guess. You can
customize it by importing the 'sys' module and changing the values of
sys.ps1 and sys.ps2.

>>> import sys
>>> sys.ps1
'>>> '
>>> sys.ps2
'... '
>>> def foo():
...     pass
...
>>> sys.ps1 = 'EGGS '
EGGS sys.ps2 = 'BACON '
EGGS def foo():
BACON     pass
BACON
EGGS


From bc at freeuk.com  Fri Dec 30 13:51:06 2016
From: bc at freeuk.com (BartC)
Date: Fri, 30 Dec 2016 18:51:06 +0000
Subject: learning and experimenting python.
In-Reply-To: <mailman.39.1483123507.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
Message-ID: <o46a81$2q4$1@dont-email.me>

On 30/12/2016 18:44, Ian Kelly wrote:
> On Fri, Dec 30, 2016 at 11:50 AM,  <einstein1410 at gmail.com> wrote:
>> Hello everyone,
>> I am the new comer and learner of python.
>> I have a doubt that when I type python and press enter it shows a prompt like
>>>>>
>> But why it is >>> ?
>> Is there any special reason?
>> Why it is not setted as @,& or any other special characters?
>
> Because it's easy to recognize as a prompt, I would guess. You can
> customize it by importing the 'sys' module and changing the values of
> sys.ps1 and sys.ps2.
>
>>>> import sys

In usenet posts >>> gets confused for quoted material 3 levels deep 
(which on my newsreader is shown as coloured vertical bars on the left).

-- 
Bartc


From einstein1410 at gmail.com  Fri Dec 30 14:08:30 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 11:08:30 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.39.1483123507.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
Message-ID: <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>

LAN you are right. I am agree with you that it's easy to recognise.

But look
$ for normal user
# for special user/root
% for other shell
>>> For python 
And so on...
Why?
Why their developer selected that?
Is there any special reason?


From einstein1410 at gmail.com  Fri Dec 30 14:09:06 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 11:09:06 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <o46a81$2q4$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <o46a81$2q4$1@dont-email.me>
Message-ID: <4cab52ec-cefc-46b0-83ba-faa2a8de328d@googlegroups.com>

So what bartc?


From none at invalid.com  Fri Dec 30 14:11:41 2016
From: none at invalid.com (mm0fmf)
Date: Fri, 30 Dec 2016 19:11:41 +0000
Subject: learning and experimenting python.
In-Reply-To: <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
Message-ID: <o46beo$76k$1@dont-email.me>

On 30/12/2016 19:08, einstein1410 at gmail.com wrote:
> LAN you are right. I am agree with you that it's easy to recognise.
>
> But look
> $ for normal user
> # for special user/root
> % for other shell
>>>> For python
> And so on...
> Why?
> Why their developer selected that?
> Is there any special reason?
>
It is what it is. Change it if you don't like it. But you will only 
confuse people who have lots of experience looking for ">>>" when you 
post some output and ask for help.


From bc at freeuk.com  Fri Dec 30 14:18:58 2016
From: bc at freeuk.com (BartC)
Date: Fri, 30 Dec 2016 19:18:58 +0000
Subject: learning and experimenting python.
In-Reply-To: <4cab52ec-cefc-46b0-83ba-faa2a8de328d@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <o46a81$2q4$1@dont-email.me>
 <4cab52ec-cefc-46b0-83ba-faa2a8de328d@googlegroups.com>
Message-ID: <o46bs9$8jg$1@dont-email.me>

On 30/12/2016 19:09, einstein1410 at gmail.com wrote:
> So what bartc?

What until you have to try and figure out which is which!

It's a disadvantage of using >>>




From best_lay at yahoo.com  Fri Dec 30 14:26:20 2016
From: best_lay at yahoo.com (Wildman)
Date: Fri, 30 Dec 2016 13:26:20 -0600
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
Message-ID: <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>

On Fri, 30 Dec 2016 11:08:30 -0800, einstein1410 wrote:

> LAN you are right. I am agree with you that it's easy to recognise.
> 
> But look
> $ for normal user
> # for special user/root
> % for other shell
>>>> For python 
> And so on...
> Why?
> Why their developer selected that?
> Is there any special reason?

That would be questions for the developer(s).  As an end-user,
the only answer I can give is, it is that way because that is
the way it is.

-- 
<Wildman> GNU/Linux user #557453
Keyboard not detected! Press any key to continue...


From jsf80238 at gmail.com  Fri Dec 30 14:37:46 2016
From: jsf80238 at gmail.com (Jason Friedman)
Date: Fri, 30 Dec 2016 12:37:46 -0700
Subject: List comprehension
Message-ID: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>

$ python
Python 3.6.0 (default, Dec 26 2016, 18:23:08)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> data = (
... (1,2),
... (3,4),
... )
>>> [a for a in data]
[(1, 2), (3, 4)]

Now, this puzzles me:

>>> [x,y for a in data]
  File "<stdin>", line 1
    [x,y for a in data]
           ^
SyntaxError: invalid syntax

I expected:
[(1, 2), (3, 4)]


From torriem at gmail.com  Fri Dec 30 14:49:22 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 30 Dec 2016 12:49:22 -0700
Subject: learning and experimenting python.
In-Reply-To: <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
Message-ID: <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>

On 12/30/2016 12:08 PM, einstein1410 at gmail.com wrote:
> LAN you are right. I am agree with you that it's easy to recognise.
> 
> But look
> $ for normal user
> # for special user/root
> % for other shell
>>>> For python 
> And so on...
> Why?
> Why their developer selected that?
> Is there any special reason?

Is there a special reason bourne shell uses $ and #?  Coming from an old
DOS background (>) I found that rather jarring at first.  There's no
particular reason for any of those shell prompts.  You say "%" is for
"other shell."  Which shells?  *Any* other shell?  These are all just
arbitrary.

Furthermore, Python is not a shell, so why would you expect an
interactive python prompt to look like bourne shell?  Wouldn't that just
be confusing?

$ python3
Python 3.4.3 (default, Aug  9 2016, 17:10:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
$

No thanks.  I think using > as a prompt character work very well. I
suppose Python could have made the prompt a bit more like ruby:

$ irb
irb(main):001:0>

But that's a bit busy.


From Joaquin.Alzola at lebara.com  Fri Dec 30 14:58:47 2016
From: Joaquin.Alzola at lebara.com (Joaquin Alzola)
Date: Fri, 30 Dec 2016 19:58:47 +0000
Subject: List comprehension
In-Reply-To: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
Message-ID: <DB5PR07MB0806BC2DE921E69DA719F99FF06A0@DB5PR07MB0806.eurprd07.prod.outlook.com>



>Now, this puzzles me:

>>>> [x,y for a in data]
>  File "<stdin>", line 1
>    [x,y for a in data]
 >          ^
>SyntaxError: invalid syntax

>I expected:
>[(1, 2), (3, 4)]

You can try [(x,z) for x,z in data].
In your situation a takes the values (1,2) or (3,4) in the one that I put x and z take the tupple values (x first one z second one).

>>> [(x,z) for x,z in data]
[(1, 2), (3, 4)]
This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.


From rosuav at gmail.com  Fri Dec 30 15:23:23 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 31 Dec 2016 07:23:23 +1100
Subject: learning and experimenting python.
In-Reply-To: <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
Message-ID: <CAPTjJmpfagN03Qc3Dt9ozswMrJ=y9ueEsT0b3o6G6uwxJOx1wA@mail.gmail.com>

On Sat, Dec 31, 2016 at 6:49 AM, Michael Torrie <torriem at gmail.com> wrote:
> Is there a special reason bourne shell uses $ and #?  Coming from an old
> DOS background (>) I found that rather jarring at first.  There's no
> particular reason for any of those shell prompts.  You say "%" is for
> "other shell."  Which shells?  *Any* other shell?  These are all just
> arbitrary.
>
> Furthermore, Python is not a shell, so why would you expect an
> interactive python prompt to look like bourne shell?  Wouldn't that just
> be confusing?
>
> $ python3
> Python 3.4.3 (default, Aug  9 2016, 17:10:39)
> [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> $
>
> No thanks.  I think using > as a prompt character work very well. I
> suppose Python could have made the prompt a bit more like ruby:

Quickly running through the interactive shells and command
interpreters I have available yields this data:

rosuav at sikorsky:~$ bash (my normal terminal prompt)
$ sh, dash
sikorsky% zsh
>>> python, python3, jython
>>>> pypy
> pike, node, threshold, vlc
rosuav=> psql
irb(main):001:0> irb (ruby)
sqlite> sqlite3
(no prompt) php, telnet, bc

And that's not counting the continuation prompts or other-mode prompts
for each of them (eg # for superuser). There are a lot of prompts, but
as you see, ">" is somewhat overused. Without even trying hard, I
found *four* programs that use it unadorned (plus I know bash uses
that as a continuation prompt), yet with the same effort, I found only
*three* with no prompt whatsoever, and they're all in "batch mode" or
similar. (Telnet, for instance, just passes everything on to the
server, so if you're talking to an SMTP server, you get no prompt.) So
there's plenty of good reason to avoid ">", just so people know what's
going on. Can you tell, for instance, what this is?

> 1+2;
3

At least this one has some adornment on its results (since you can
retrieve prior results by index):

> 1+2;
(1) Result: 3

Those are the Node.js and Pike REPLs, respectively. Python has an
advantage because you can instantly see that a transcript comes from
Python. :)

ChrisA


From joel.goldstick at gmail.com  Fri Dec 30 15:25:53 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Fri, 30 Dec 2016 15:25:53 -0500
Subject: List comprehension
In-Reply-To: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
Message-ID: <CAPM-O+zxqon032e1wNw=aOWYdCS-49jv2hkP_Dnzoo464c1o-A@mail.gmail.com>

On Fri, Dec 30, 2016 at 2:37 PM, Jason Friedman <jsf80238 at gmail.com> wrote:
> $ python
> Python 3.6.0 (default, Dec 26 2016, 18:23:08)
> [GCC 4.8.4] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> data = (
> ... (1,2),
> ... (3,4),
> ... )
>>>> [a for a in data]
> [(1, 2), (3, 4)]
>
> Now, this puzzles me:
>
>>>> [x,y for a in data]
>   File "<stdin>", line 1
>     [x,y for a in data]
>            ^
> SyntaxError: invalid syntax
>
> I expected:
> [(1, 2), (3, 4)]
> --
> https://mail.python.org/mailman/listinfo/python-list

Thy this:

>>> [(a[0], a[1]) for a in data]
[(1, 2), (3, 4)]


-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From david.froger.ml at mailoo.org  Fri Dec 30 15:46:29 2016
From: david.froger.ml at mailoo.org (David Froger)
Date: Fri, 30 Dec 2016 21:46:29 +0100
Subject: learning and experimenting python.
In-Reply-To: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
Message-ID: <148313078934.2259.7054529080896733101@mael>

Quoting einstein1410 at gmail.com (2016-12-30 18:50:19)
> Hello everyone,
> I am the new comer and learner of python.
> I have a doubt that when I type python and press enter it shows a prompt like
> >>>
> But why it is >>> ?
> Is there any special reason?
> Why it is not setted as @,& or any other special characters?
> -- 
> https://mail.python.org/mailman/listinfo/python-list

Seems that it is inherited from the ABC programming langage:
    http://homepages.cwi.nl/~steven/abc/types.html

The question is now why ABC choose >>> as prompt...


From python at deborahswanson.net  Fri Dec 30 16:20:15 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 13:20:15 -0800
Subject: Cleaning up conditionals
In-Reply-To: <7a3c695c-7d20-4227-8e95-e5e3ebce359d@googlegroups.com>
Message-ID: <000401d262e2$83f6b480$27b23dae@sambora>

I've already learned one neat trick to collapse a conditional:

	a = expression1 if condition else expression2

Here I have a real mess, in my opinion:

	if len(l1[st]) == 0:
          if len(l2[st]) > 0:
              l1[st] = l2[st]
          elif len(l2[st]) == 0:
              if len(l1[st]) > 0:
                  l2[st] = l1[st]

  (Basically, if one field from two adjacent rows is empty and the other
is 
  not, copy the non-empty field to the empty one. I use this for rental 
  listings that are identical but posted on different dates, to copy the

  data from an older one to the new one. Or, if I look up the data for a
new  
  listing, to copy it back to the older ones.)

Anybody know or see an easier (more pythonic) way to do this? I need to
do it for four fields, and needless to say, that's a really long block
of ugly code.

Any other insights into tightening up complex conditionals?

Thanks in advance,
Deborah 



From bc at freeuk.com  Fri Dec 30 17:11:16 2016
From: bc at freeuk.com (BartC)
Date: Fri, 30 Dec 2016 22:11:16 +0000
Subject: Cleaning up conditionals
In-Reply-To: <mailman.46.1483135377.2395.python-list@python.org>
References: <7a3c695c-7d20-4227-8e95-e5e3ebce359d@googlegroups.com>
 <000401d262e2$83f6b480$27b23dae@sambora>
 <mailman.46.1483135377.2395.python-list@python.org>
Message-ID: <o46lvc$hbt$1@dont-email.me>

On 30/12/2016 21:20, Deborah Swanson wrote:
> I've already learned one neat trick to collapse a conditional:
>
> 	a = expression1 if condition else expression2
>
> Here I have a real mess, in my opinion:
>
> 	if len(l1[st]) == 0:
>           if len(l2[st]) > 0:
>               l1[st] = l2[st]
>           elif len(l2[st]) == 0:
>               if len(l1[st]) > 0:
>                   l2[st] = l1[st]

This doesn't make sense. The main block is executed when len(l1[st]) is 
0, but you're testing for len(l1[st])>0 in the last if statement (which 
can't see the assignment to l1[st], so can never be true).

Try writing it out on paper using A and B instead l1[st] and l2[st] as 
they look confusing.

You might also be evaluating len(l2[st]) twice.

-- 
Bartc


From joel.goldstick at gmail.com  Fri Dec 30 17:30:25 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Fri, 30 Dec 2016 17:30:25 -0500
Subject: List comprehension
In-Reply-To: <DB5PR07MB0806BC2DE921E69DA719F99FF06A0@DB5PR07MB0806.eurprd07.prod.outlook.com>
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
 <DB5PR07MB0806BC2DE921E69DA719F99FF06A0@DB5PR07MB0806.eurprd07.prod.outlook.com>
Message-ID: <CAPM-O+wTcHywUyX4JBz2mPtXAM49GFTmTHJfGGNLdbA-cZ6K+w@mail.gmail.com>

On Fri, Dec 30, 2016 at 2:58 PM, Joaquin Alzola
<Joaquin.Alzola at lebara.com> wrote:
>
>
>>Now, this puzzles me:
>
>>>>> [x,y for a in data]
>>  File "<stdin>", line 1
>>    [x,y for a in data]
>  >          ^
>>SyntaxError: invalid syntax
>
>>I expected:
>>[(1, 2), (3, 4)]
>
> You can try [(x,z) for x,z in data].
> In your situation a takes the values (1,2) or (3,4) in the one that I put x and z take the tupple values (x first one z second one).
>
>>>> [(x,z) for x,z in data]
> [(1, 2), (3, 4)]

I like yours better than mine.


-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From python at deborahswanson.net  Fri Dec 30 18:00:02 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 15:00:02 -0800
Subject: Cleaning up conditionals
In-Reply-To: <o46lvc$hbt$1@dont-email.me>
Message-ID: <001a01d262f0$74b2be70$27b23dae@sambora>

BartC wrote:
> Sent: Friday, December 30, 2016 2:11 PM
> 
> On 30/12/2016 21:20, Deborah Swanson wrote:
> > I've already learned one neat trick to collapse a conditional:
> >
> > 	a = expression1 if condition else expression2
> >
> > Here I have a real mess, in my opinion:
> >
> > 	if len(l1[st]) == 0:
> >           if len(l2[st]) > 0:
> >               l1[st] = l2[st]
> >           elif len(l2[st]) == 0:
> >               if len(l1[st]) > 0:
> >                   l2[st] = l1[st]
> 
> This doesn't make sense. The main block is executed when 
> len(l1[st]) is 
> 0, but you're testing for len(l1[st])>0 in the last if 
> statement (which 
> can't see the assignment to l1[st], so can never be true).
> Try writing it out on paper using A and B instead l1[st] and 
> l2[st] as 
> they look confusing.
> 
> You might also be evaluating len(l2[st]) twice.
> 
> -- 
> Bartc

Oops, indentation was messed up when I copied it into the email. Should
be this:

		if len(l1[st]) == 0:
                if len(l2[st]) > 0:
                    l1[st] = l2[st]
            elif len(l2[st]) == 0:
                if len(l1[st]) > 0:
                    l2[st] = l1[st]



From bouncingcats at gmail.com  Fri Dec 30 18:16:38 2016
From: bouncingcats at gmail.com (David)
Date: Sat, 31 Dec 2016 10:16:38 +1100
Subject: Cleaning up conditionals
In-Reply-To: <001a01d262f0$74b2be70$27b23dae@sambora>
References: <o46lvc$hbt$1@dont-email.me>
 <001a01d262f0$74b2be70$27b23dae@sambora>
Message-ID: <CAMPXz=rb=D8pMBa7if4ZXMd9YN4KHtT+0y+mTy1WLDgEdhAH=A@mail.gmail.com>

On 31 December 2016 at 10:00, Deborah Swanson <python at deborahswanson.net> wrote:
>
> Oops, indentation was messed up when I copied it into the email.

The indentation of your latest message looks completely broken  now,
you can see it here:
https://mail.python.org/pipermail/python-list/2016-December/717758.html


From python at deborahswanson.net  Fri Dec 30 18:17:55 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 15:17:55 -0800
Subject: Cleaning up conditionals
In-Reply-To: <4jnd6cd7qk7uc49oldbttkr6jnnfqq3gvi@4ax.com>
Message-ID: <002401d262f2$f429bf30$27b23dae@sambora>

> On Fri, 30 Dec 2016 13:20:15 -0800, "Deborah Swanson" 
> <python at deborahswanson.net> declaimed the following:
> 
> >I've already learned one neat trick to collapse a conditional:
> >
> >	a = expression1 if condition else expression2
> >
> >Here I have a real mess, in my opinion:
> >
> >	if len(l1[st]) == 0:
> >          if len(l2[st]) > 0:
> >              l1[st] = l2[st]
> >          elif len(l2[st]) == 0:
> >              if len(l1[st]) > 0:
> 
> 	You will never reach here. The above "elif" is only 
> reachable if "len(l1[st])" IS EQUAL TO 0, which means that 
> the later "if" can not be true.
> 
> >                  l2[st] = l1[st]
> >
> >  (Basically, if one field from two adjacent rows is empty and the 
> >other is
> >  not, copy the non-empty field to the empty one. I use this 
> for rental
> >  listings that are identical but posted on different dates, 
> to copy the
> >
> >  data from an older one to the new one. Or, if I look up 
> the data for 
> >a new
> >  listing, to copy it back to the older ones.)
> >
> >Anybody know or see an easier (more pythonic) way to do 
> this? I need to 
> >do it for four fields, and needless to say, that's a really 
> long block 
> >of ugly code.
> >
> 
> 	Ever consider using conjunctions?
> 
> 	if len(l1[st]) and not len(l2[st]):
> 		#0 is considered a false -- no need to test for "==0"
> 		#non-0 is considered true -- no need to test for ">0"
> 		#copy l1 to l2
> 	elif not len(l1[st]) and len(l2[st]):
> 		#copy l2 to l1
> 
> -- 
> 	Wulfraed                 Dennis Lee Bieber         AF6VN
>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/

That's a neat shortcut, len(a) instead of len(a)!= 0.  Thanks!

Yes, 4 lines is an improvement on 6 lines, but I was hoping for
something more radical.

Is it possible to use some version of the "a = expression1 if condition
else expression2" syntax with an elif? And for expression1 and
expression2 to be single statements?  That's the kind of shortcutting
I'd like to do, and it seems like python might be able to do something
like this.



From python at lucidity.plus.com  Fri Dec 30 18:22:02 2016
From: python at lucidity.plus.com (Erik)
Date: Fri, 30 Dec 2016 23:22:02 +0000
Subject: Cleaning up conditionals
In-Reply-To: <001a01d262f0$74b2be70$27b23dae@sambora>
References: <001a01d262f0$74b2be70$27b23dae@sambora>
Message-ID: <607116a6-dc08-ea89-504a-ebd32576ba97@lucidity.plus.com>

On 30/12/16 23:00, Deborah Swanson wrote:
> Oops, indentation was messed up when I copied it into the email. Should
> be this:
>
> 		if len(l1[st]) == 0:
>                 if len(l2[st]) > 0:
>                     l1[st] = l2[st]
>             elif len(l2[st]) == 0:
>                 if len(l1[st]) > 0:
>                     l2[st] = l1[st]

That's even worse!

Anyway, ignoring all that, if what you are trying to do is just do some 
action based on a set of fixed comparisons that are known at the top of 
the function (and do not mutate depending on the path through the 
function), then you can just cache the comparisons and then compare the 
resulting set of boolean results (note that my examples are NOT based on 
your use-case, it's just pseudo-code which happens to use your expressions):

state = (len(l1[st]) == 0, len(l2[st]) > 0)
if state == (True, False):
   pass
elif state == (False, True):
   pass

... etc.

If the len() comparisons are tri-state (i.e., in some cases you want to 
know if the length is <, ==, or > 0, depending on one of the other 
comparisons) then you can do something like:

def clamp(foo):
   return min(max(-1, foo), 1)

state = (clamp(cmp(len(l1[st]), 0), cmp(len(l2[st]), 0))
if state == (0, -1):
   pass
elif state == (1, -1):
   pass

... etc.

I'm not sure this makes it much more readable though - but if you make 
the RHS of those comparisons a symbolic name you might be getting 
somewhere -

ACTION1 = (0, -1)
ACTION2 = (1, -1)
if state == ACTION1:
   pass
elif state == ACTION2:
   pass

Hope that helps, E.


From einstein1410 at gmail.com  Fri Dec 30 18:26:05 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:26:05 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <o46beo$76k$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <o46beo$76k$1@dont-email.me>
Message-ID: <96e184f9-7da9-4105-bdfa-7a0b6e7171bf@googlegroups.com>

That's not the answer.
If you don't have answer, please don't answer like this, because that will confuse others also.


From einstein1410 at gmail.com  Fri Dec 30 18:28:53 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:28:53 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <o46bs9$8jg$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <o46a81$2q4$1@dont-email.me>
 <4cab52ec-cefc-46b0-83ba-faa2a8de328d@googlegroups.com>
 <o46bs9$8jg$1@dont-email.me>
Message-ID: <7f7b33da-284a-44ca-9334-fce60680e247@googlegroups.com>

I am not getting you.
Please simplify it.


From einstein1410 at gmail.com  Fri Dec 30 18:34:16 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:34:16 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
Message-ID: <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>

You are also confusing me.
But there mustbe  some reason.
What happens if your student questions you like this.?
And may be those who questions like this will surely be the developer of its successor language.
Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius.


From einstein1410 at gmail.com  Fri Dec 30 18:38:09 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:38:09 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.41.1483127773.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
Message-ID: <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>

Yes, I am not confusing you all, rather I thought that this is the best place to solve any doubts, soy only question for you is

Why python uses >>> instead of >, or any other special characters?
Do you know about this, if yes then please answer it.
I will be so much thankful of you.


From python at lucidity.plus.com  Fri Dec 30 18:39:43 2016
From: python at lucidity.plus.com (Erik)
Date: Fri, 30 Dec 2016 23:39:43 +0000
Subject: learning and experimenting python.
In-Reply-To: <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
Message-ID: <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>

On 30/12/16 23:34, einstein1410 at gmail.com wrote:
> You are also confusing me.
> But there mustbe  some reason.
> What happens if your student questions you like this.?
> And may be those who questions like this will surely be the developer of its successor language.
> Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius

Do not feed the troll.

E.



From einstein1410 at gmail.com  Fri Dec 30 18:52:08 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:52:08 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.44.1483133004.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <148313078934.2259.7054529080896733101@mael>
 <mailman.44.1483133004.2395.python-list@python.org>
Message-ID: <b1d657bc-a740-4e5f-a740-efd1b19e9c7b@googlegroups.com>

Then again the question is same for ABC.
Why >>>?


From einstein1410 at gmail.com  Fri Dec 30 18:56:18 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 15:56:18 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.42.1483129412.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <CAPTjJmpfagN03Qc3Dt9ozswMrJ=y9ueEsT0b3o6G6uwxJOx1wA@mail.gmail.com>
 <mailman.42.1483129412.2395.python-list@python.org>
Message-ID: <841c158a-94d2-4c7f-b0ed-936b2ba4b374@googlegroups.com>

No I have a question not answer, but if I got the answer then I will tell you.
Thanks.


From ian.g.kelly at gmail.com  Fri Dec 30 19:01:14 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 30 Dec 2016 18:01:14 -0600
Subject: learning and experimenting python.
In-Reply-To: <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
Message-ID: <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>

On Dec 30, 2016 4:42 PM, <einstein1410 at gmail.com> wrote:

Yes, I am not confusing you all, rather I thought that this is the best
place to solve any doubts, soy only question for you is

Why python uses >>> instead of >, or any other special characters?
Do you know about this, if yes then please answer it.
I will be so much thankful of you.


The answers you've already received are the best you're likely to get here.
Nobody knows exactly why because the question is about unimportant trivia.
If you really want to know you could ask Guido (if he even remembers the
reason) and then you can be the expert.


From einstein1410 at gmail.com  Fri Dec 30 19:03:22 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 16:03:22 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.53.1483141372.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
Message-ID: <724cde29-bbb1-4a7a-8675-616886a3a5f1@googlegroups.com>

Im not feeding the troll.
If you don't have answer to this, then there would be no question that you are the python developer.
And if this is troll, leave it.
But respect every question, they are the gift of God. And if there is no question, there will be no answer. And may the question is the answer to this giant development.
If no question arises in the person who is now recognised as the developer, then now a days he will not be that what he is...


From einstein1410 at gmail.com  Fri Dec 30 19:05:25 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 16:05:25 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.54.1483142484.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
Message-ID: <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>

But in his website, he recommended that post your questions here, he will answer it.
But still as you told me I will send him an personal e-mail.


From einstein1410 at gmail.com  Fri Dec 30 19:08:01 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 16:08:01 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.54.1483142484.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
Message-ID: <92b66c21-bf60-4926-b747-140d1d2e2199@googlegroups.com>

And if this is unimportant, as you thought, then you must not be the part of this group. This group is to share knowledge, nott for deciding what is trivia and what's not.
If you think it's it, simply leave it, any else will answer.


From python at deborahswanson.net  Fri Dec 30 19:26:43 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 16:26:43 -0800
Subject: Cleaning up conditionals
In-Reply-To: <CAMPXz=rb=D8pMBa7if4ZXMd9YN4KHtT+0y+mTy1WLDgEdhAH=A@mail.gmail.com>
Message-ID: <003501d262fc$90a70030$27b23dae@sambora>

> On 31 December 2016 at 10:00, Deborah Swanson 
> <python at deborahswanson.net> wrote:
> >
> > Oops, indentation was messed up when I copied it into the email.
> 
> The indentation of your latest message looks completely 
> broken  now, you can see it here: 
> https://mail.python.org/pipermail/python-list/2016-December/71
7758.html

Hm, I don't know what pipermail does with whitespace but the formatting
in the message at
https://mail.python.org/pipermail/python-list/2016-December/717758.html
is not the same as the message I sent from Outlook.

Again, it should have been:

if len(l1[st]) == 0:
	if len(l2[st]) > 0:
		l1[st] = l2[st]
elif len(l2[st]) == 0:
	if len(l1[st]) > 0:
		l2[st] = l1[st]

(Maybe keeping the left edge flush with the left margin will help keep
the mail processors straight. I've never tried emailing python code
before.)

As Mr. Bieber points out, what I had above greatly benefits from the use
of conjunctions. It now reads:

if not len(l1[st]) and len(l2[st]):
		l1[st] = l2[st]
elif not len(l2[st]) and len(l1[st]):
		l2[st] = l1[st]

(I plead being cross-eyed from rewriting this section so many times for
not seeing this.)

I'm still wondering if these 4 lines can be collapsed to one or two
lines.



From steve+python at pearwood.info  Fri Dec 30 19:30:26 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 11:30:26 +1100
Subject: Simulating int arithmetic with wrap-around
References: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
 <o45tji$fc7$1@blaine.gmane.org>
 <mailman.34.1483110855.2395.python-list@python.org>
Message-ID: <5866fc24$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 02:14 am, Serhiy Storchaka wrote:

> On 30.12.16 16:47, Steve D'Aprano wrote:
>> Again, assume both operands are in range for an N-bit signed integer.
>> What's a good way to efficiently, or at least not too inefficiently, do
>> the calculations in Python?
> 
> def to_unsigned(bits, x):
>      return x & ((1<<bits)-1)
> 
> def to_signed(bits, x):
>      offset = 1<<(bits-1)
>      return to_unsigned(bits, x + offset) - offset

Thanks.

Are you saying that the best way of doing this is this?

(1) convert signed Python ints to unsigned;

(2) perform operation and bitmask;

(3) convert unsigned back to signed.


Here's an example. I want to multiply 7*3 using a signed 4-bit int, getting
5 as the answer, and 7*4 getting -4 as the answer:


py> N = 4
py> to_signed(N, to_unsigned(N, 7) * to_unsigned(N, 3) & (2**N - 1))
5
py> to_signed(N, to_unsigned(N, 7) * to_unsigned(N, 4) & (2**N - 1))
-4



It works, but I'm glad I won't be doing anything that requires great
speed :-)




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Fri Dec 30 19:37:55 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 11:37:55 +1100
Subject: List comprehension
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
 <mailman.40.1483126677.2395.python-list@python.org>
Message-ID: <5866fde5$0$1609$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 06:37 am, Jason Friedman wrote:

> $ python
> Python 3.6.0 (default, Dec 26 2016, 18:23:08)
> [GCC 4.8.4] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> data = (
> ... (1,2),
> ... (3,4),
> ... )
>>>> [a for a in data]
> [(1, 2), (3, 4)]
> 
> Now, this puzzles me:
> 
>>>> [x,y for a in data]
>   File "<stdin>", line 1
>     [x,y for a in data]
>            ^
> SyntaxError: invalid syntax
> 
> I expected:
> [(1, 2), (3, 4)]


Why would you expect that? I would expect the global variables x and y, or
if they don't exist, a NameError:

py> a = (1, 2)
py> x, y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

Python has no magic that "x and y mean the first two items of a".


Instead, you get a syntax error because the syntax is ambiguous:

[x,y for a in data]

looks like a list

[x, y]

except the second item is a syntax error "y for a in data". So let's fix the
syntax error:



py> data = [(1, 2), (3, 4)]
py> [a for a in data]
[(1, 2), (3, 4)]
py> [(x, y) for a in data]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
NameError: name 'x' is not defined


Okay, let's create global variables x and y:

py> x = 98
py> y = 99
py> [(x, y) for a in data]
[(98, 99), (98, 99)]


No surprises there. How about this instead:

py> [(x, y) for (x, y) in data]
[(1, 2), (3, 4)]


That's better!




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From python at lucidity.plus.com  Fri Dec 30 19:43:28 2016
From: python at lucidity.plus.com (Erik)
Date: Sat, 31 Dec 2016 00:43:28 +0000
Subject: Cleaning up conditionals
In-Reply-To: <003501d262fc$90a70030$27b23dae@sambora>
References: <003501d262fc$90a70030$27b23dae@sambora>
Message-ID: <bece301a-293d-5639-91c3-1d008dfcc1f0@lucidity.plus.com>

On 31/12/16 00:26, Deborah Swanson wrote:
> As Mr. Bieber points out, what I had above greatly benefits from the use
> of conjunctions. It now reads:
>
> if not len(l1[st]) and len(l2[st]):

IMHO, "if not len(l)" is a _terrible_ way of spelling "if len(l) == 0" 
(mentally, I have to read that as "if length of 'l' is not not equal to 
0" - and a double negative won't never cause problems ( ;) )).

Also, in that particular expression, having to know off the top of their 
head the precedence of 'not' and 'and' will cause at least some 
percentage of your maintenance audience in the future to get it wrong.

What's wrong with:

if len(l1[st]) == 0 and len(l2[st]) != 0:
   ...

?

There is _no way_ someone could read that and get the wrong idea.

E.


From steve+python at pearwood.info  Fri Dec 30 19:44:48 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 11:44:48 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <o46a81$2q4$1@dont-email.me>
Message-ID: <5866ff82$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 05:51 am, BartC wrote:

> In usenet posts >>> gets confused for quoted material 3 levels deep
> (which on my newsreader is shown as coloured vertical bars on the left).


Indeed, which is why my Python startup file contains this:



# Change the main prompt. Consider using '??? '?
# This is non-portable and may not work everywhere.
if (sys.version_info[0] >= 3 and os.name == 'posix'
    and os.environ['TERM'] in ['xterm', 'vt100']):
    # Make the prompt bold in Python 3.
    sys.ps1 = '\001\x1b[1m\002py> \001\x1b[0m\002'
    sys.ps2 = '\001\x1b[1m\002... \001\x1b[0m\002'
else:
    sys.ps1 = 'py> '


That gives me a "py>" prompt, bolded in Python 3 and plain in Python 2.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From python at deborahswanson.net  Fri Dec 30 19:48:16 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 16:48:16 -0800
Subject: Cleaning up conditionals
In-Reply-To: <607116a6-dc08-ea89-504a-ebd32576ba97@lucidity.plus.com>
Message-ID: <003801d262ff$931dec40$27b23dae@sambora>

> On 30/12/16 23:00, Deborah Swanson wrote:
> > Oops, indentation was messed up when I copied it into the email. 
> > Should be this:
> >
> > 		if len(l1[st]) == 0:
> >                 if len(l2[st]) > 0:
> >                     l1[st] = l2[st]
> >             elif len(l2[st]) == 0:
> >                 if len(l1[st]) > 0:
> >                     l2[st] = l1[st]
> 
> That's even worse!
> 
> Anyway, ignoring all that, if what you are trying to do is 
> just do some 
> action based on a set of fixed comparisons that are known at 
> the top of 
> the function (and do not mutate depending on the path through the 
> function), then you can just cache the comparisons and then 
> compare the 
> resulting set of boolean results (note that my examples are 
> NOT based on 
> your use-case, it's just pseudo-code which happens to use 
> your expressions):
> 
> state = (len(l1[st]) == 0, len(l2[st]) > 0)
> if state == (True, False):
>    pass
> elif state == (False, True):
>    pass
> 
> ... etc.
> 
> If the len() comparisons are tri-state (i.e., in some cases 
> you want to 
> know if the length is <, ==, or > 0, depending on one of the other 
> comparisons) then you can do something like:
> 
> def clamp(foo):
>    return min(max(-1, foo), 1)
> 
> state = (clamp(cmp(len(l1[st]), 0), cmp(len(l2[st]), 0))
> if state == (0, -1):
>    pass
> elif state == (1, -1):
>    pass
> 
> ... etc.
> 
> I'm not sure this makes it much more readable though - but if 
> you make 
> the RHS of those comparisons a symbolic name you might be getting 
> somewhere -
> 
> ACTION1 = (0, -1)
> ACTION2 = (1, -1)
> if state == ACTION1:
>    pass
> elif state == ACTION2:
>    pass
> 
> Hope that helps, E.
> 

Thanks Erik, I really like your method of capturing the state and then
reusing it. I'm sure I can use it in other situations. In this case,
since I now have it down to:

if not len(l1[st]) and len(l2[st]):
		l1[st] = l2[st]
elif not len(l2[st]) and len(l1[st]):
		l2[st] = l1[st]

I'm not sure caching the state really simplifies that much, although I'm
going to play around with caching the state of all four fields for both
rows, or maybe for all rows to be compared. The code I've been posting
is only for one field and 2 rows, but there are 4 fields to be compared
and there can be up to 10 rows (or more) that are identical except for
the date. Would be nifty to do them all in one swipe. 

It's possible that there are some efficiencies to be found in treating
the 4 field comparisons as one problem. Or maybe I should write a little
function, since the only thing that changes in the comparisons for a set
of rows is the 4 fields. I'm going to try that now, and quite possibly
caching the state would really be useful in that function. Just gobble
them all up in one bite!



From python at deborahswanson.net  Fri Dec 30 19:54:29 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 16:54:29 -0800
Subject: Cleaning up conditionals
In-Reply-To: <bece301a-293d-5639-91c3-1d008dfcc1f0@lucidity.plus.com>
Message-ID: <003901d26300$71b8c510$27b23dae@sambora>

> On 31/12/16 00:26, Deborah Swanson wrote:
> > As Mr. Bieber points out, what I had above greatly benefits 
> from the 
> > use of conjunctions. It now reads:
> >
> > if not len(l1[st]) and len(l2[st]):
> 
> IMHO, "if not len(l)" is a _terrible_ way of spelling "if 
> len(l) == 0" 
> (mentally, I have to read that as "if length of 'l' is not 
> not equal to 
> 0" - and a double negative won't never cause problems ( ;) )).
> 
> Also, in that particular expression, having to know off the 
> top of their 
> head the precedence of 'not' and 'and' will cause at least some 
> percentage of your maintenance audience in the future to get it wrong.
> 
> What's wrong with:
> 
> if len(l1[st]) == 0 and len(l2[st]) != 0:
>    ...
> 
> ?

Absolutely nothing wrong, and you're right that it's more readable. I
just think it's cool that Python will do the right thing with not len(a)
and len(a).
 
> There is _no way_ someone could read that and get the wrong idea.
> 
> E.

Quite true.



From storchaka at gmail.com  Fri Dec 30 20:11:49 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 31 Dec 2016 03:11:49 +0200
Subject: Simulating int arithmetic with wrap-around
In-Reply-To: <5866fc24$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <58667385$0$1609$c3e8da3$5496439d@news.astraweb.com>
 <o45tji$fc7$1@blaine.gmane.org>
 <mailman.34.1483110855.2395.python-list@python.org>
 <5866fc24$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o470kg$99c$1@blaine.gmane.org>

On 31.12.16 02:30, Steve D'Aprano wrote:
> Are you saying that the best way of doing this is this?
>
> (1) convert signed Python ints to unsigned;
>
> (2) perform operation and bitmask;
>
> (3) convert unsigned back to signed.
>
>
> Here's an example. I want to multiply 7*3 using a signed 4-bit int, getting
> 5 as the answer, and 7*4 getting -4 as the answer:
>
>
> py> N = 4
> py> to_signed(N, to_unsigned(N, 7) * to_unsigned(N, 3) & (2**N - 1))
> 5
> py> to_signed(N, to_unsigned(N, 7) * to_unsigned(N, 4) & (2**N - 1))
> -4

Step 1 is not needed. And you can inline to_unsigned() in to_signed(). I 
introduced it just for clearness.

py> to_signed(N, 7 * 3)
5
py> to_signed(N, 7 * 4)
-4




From eryksun at gmail.com  Fri Dec 30 20:46:25 2016
From: eryksun at gmail.com (eryk sun)
Date: Sat, 31 Dec 2016 01:46:25 +0000
Subject: learning and experimenting python.
In-Reply-To: <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
Message-ID: <CACL+1assa3vawJhHMgG1_FALLCXJEZKOHs2RmJu3AEjgk5ESEQ@mail.gmail.com>

On Fri, Dec 30, 2016 at 7:49 PM, Michael Torrie <torriem at gmail.com> wrote:
>
> Is there a special reason bourne shell uses $ and #?

To me, "$"  is for the [$]tandard shell prompt, and "#" noticeably
distinguishes root shells.

> Coming from an old DOS background (>) I found that rather jarring at first.

DOS is a single-user system with no security, so it doesn't need to
distinguish standard and root prompts.

In modern Windows, cmd and PowerShell change the console title instead
of the prompt. The title starts with "Administrator: " if the user is
an administrator (i.e. the BUILTIN\Administrators group is present and
enabled in the process token).

> You say "%" is for "other shell."  Which shells?  *Any* other shell?

"%" is the standard prompt for csh. I think the legacy Thompson shell
also used it.


From best_lay at yahoo.com  Fri Dec 30 20:57:16 2016
From: best_lay at yahoo.com (Wildman)
Date: Fri, 30 Dec 2016 19:57:16 -0600
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
Message-ID: <MLSdnUNcZ8rhjfrFnZ2dnUU7-YWdnZ2d@giganews.com>

On Fri, 30 Dec 2016 15:34:16 -0800, einstein1410 wrote:

> You are also confusing me.
> But there mustbe  some reason.
> What happens if your student questions you like this.?

I am not a teacher.

> And may be those who questions like this will surely be the developer of its successor language.
> Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius.

To get an answer to your questions you need to ask the developer.  I don't
think I can get any clearer.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From cs at zip.com.au  Fri Dec 30 20:59:30 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 31 Dec 2016 12:59:30 +1100
Subject: Cleaning up conditionals
In-Reply-To: <002401d262f2$f429bf30$27b23dae@sambora>
References: <002401d262f2$f429bf30$27b23dae@sambora>
Message-ID: <20161231015930.GA73163@cskk.homeip.net>

On 30Dec2016 15:17, Deborah Swanson <python at deborahswanson.net> wrote:
>> 	Ever consider using conjunctions?
>>
>> 	if len(l1[st]) and not len(l2[st]):
>> 		#0 is considered a false -- no need to test for "==0"
>> 		#non-0 is considered true -- no need to test for ">0"
>> 		#copy l1 to l2
>> 	elif not len(l1[st]) and len(l2[st]):
>> 		#copy l2 to l1
>> --
>> 	Wulfraed                 Dennis Lee Bieber         AF6VN
>>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/
>
>That's a neat shortcut, len(a) instead of len(a)!= 0.  Thanks!

Also, for almost every python collection (lists, tuples, sets etc), python 
boolean logic tests __nonzero__, which works off len() by default.

So:

  if a:
    # a is not empty: len(a) > 0
  else:
    # a is empty: len(a) == 0

I find this far more readable, presuming the reader knows that "empty" things 
test as false. Of course, you need to ensure that any "collection"-ish classes 
you use or write have this property, but the builtin ones do.

Cheers,
Cameron Simpson <cs at zip.com.au>


From best_lay at yahoo.com  Fri Dec 30 21:05:53 2016
From: best_lay at yahoo.com (Wildman)
Date: Fri, 30 Dec 2016 20:05:53 -0600
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
Message-ID: <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>

On Fri, 30 Dec 2016 23:39:43 +0000, Erik wrote:

> On 30/12/16 23:34, einstein1410 at gmail.com wrote:
>> You are also confusing me.
>> But there mustbe  some reason.
>> What happens if your student questions you like this.?
>> And may be those who questions like this will surely be the developer of its successor language.
>> Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius
> 
> Do not feed the troll.
> 
> E.

Please explain how what I said is trolling.  Perhaps it was a little
snide but I tend to get that way when trying to explain the obvious.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From torriem at gmail.com  Fri Dec 30 21:10:07 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 30 Dec 2016 19:10:07 -0700
Subject: learning and experimenting python.
In-Reply-To: <CACL+1assa3vawJhHMgG1_FALLCXJEZKOHs2RmJu3AEjgk5ESEQ@mail.gmail.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <CACL+1assa3vawJhHMgG1_FALLCXJEZKOHs2RmJu3AEjgk5ESEQ@mail.gmail.com>
Message-ID: <2c0749c1-a735-e3c0-b423-c9f88f782e37@gmail.com>

On 12/30/2016 06:46 PM, eryk sun wrote:
> On Fri, Dec 30, 2016 at 7:49 PM, Michael Torrie <torriem at gmail.com> wrote:
>>
>> Is there a special reason bourne shell uses $ and #?
> 
> To me, "$"  is for the [$]tandard shell prompt, and "#" noticeably
> distinguishes root shells.

Yes of course. You missed my point completely.  These prompts have
meaning only because they were picked arbitrarily.  Bourne shell could
well have used # for user prompt and $ for the root prompt.

>> Coming from an old DOS background (>) I found that rather jarring at first.
> 
> DOS is a single-user system with no security, so it doesn't need to
> distinguish standard and root prompts.

Again I think you missed my point. Or you read more into it than I gave.
I was not confused by the unix prompts. I understood the difference
between root and normal user.  Rather I liked the appearance of the
Windows-style prompt simply because I was used to it.

Now I prefer the bash prompt and I greatly value the different root
prompt character.


From torriem at gmail.com  Fri Dec 30 21:18:47 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 30 Dec 2016 19:18:47 -0700
Subject: learning and experimenting python.
In-Reply-To: <96e184f9-7da9-4105-bdfa-7a0b6e7171bf@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <o46beo$76k$1@dont-email.me>
 <96e184f9-7da9-4105-bdfa-7a0b6e7171bf@googlegroups.com>
Message-ID: <5562b895-c4fc-6470-a275-d2489fae1de1@gmail.com>

On 12/30/2016 04:26 PM, einstein1410 at gmail.com wrote:
> That's not the answer.
> If you don't have answer, please don't answer like this, because that will confuse others also.

I don't believe anyone will be confused.

Clearly there's no answer that you understand, or one that would satisfy
you.  There could be good reasons for choosing >>>.  Or there could be
no reason at all.  Maybe Guido liked the > but to make it a bit more
unique, made it >>>.  Maybe he had a dream and saw >>> in the sky.  Or
maybe he just got the idea from xkcd.  Why does this matter?  We've
given you a plethora of rationale and possible ideas why it is this way.
 Just pick one, any one, and there's your answer.



From torriem at gmail.com  Fri Dec 30 21:23:17 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 30 Dec 2016 19:23:17 -0700
Subject: learning and experimenting python.
In-Reply-To: <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
Message-ID: <adf2731a-5bf4-817f-6f32-a54f7de3c174@gmail.com>

On 12/30/2016 07:05 PM, Wildman via Python-list wrote:
> On Fri, 30 Dec 2016 23:39:43 +0000, Erik wrote:
> 
>> On 30/12/16 23:34, einstein1410 at gmail.com wrote:
>>> You are also confusing me.
>>> But there mustbe  some reason.
>>> What happens if your student questions you like this.?
>>> And may be those who questions like this will surely be the developer of its successor language.
>>> Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius
>>
>> Do not feed the troll.
>>
>> E.
> 
> Please explain how what I said is trolling.  Perhaps it was a little
> snide but I tend to get that way when trying to explain the obvious.

Hmm. I thought he was referring to einstein1410...  It was his message
he was replying to, not yours, and I took it as a request to the rest of
us (sorry, Erik--couldn't resist posting).



From ethan at stoneleaf.us  Fri Dec 30 21:26:08 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 30 Dec 2016 18:26:08 -0800
Subject: learning and experimenting python.
In-Reply-To: <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
Message-ID: <58671740.7000908@stoneleaf.us>

On 12/30/2016 06:05 PM, Wildman via Python-list wrote:
> On Fri, 30 Dec 2016 23:39:43 +0000, Erik wrote:

>> Do not feed the troll.
>
> Please explain how what I said is trolling.  Perhaps it was a little
> snide but I tend to get that way when trying to explain the obvious.

I suspect Erik was referring to "einstein1410", who seems to be wasting time and bandwidth on silly questions.

--
~Ethan~


From torriem at gmail.com  Fri Dec 30 21:28:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Fri, 30 Dec 2016 19:28:39 -0700
Subject: Cleaning up conditionals
In-Reply-To: <003501d262fc$90a70030$27b23dae@sambora>
References: <003501d262fc$90a70030$27b23dae@sambora>
Message-ID: <60f00215-4a75-f017-4670-0cd794593dc6@gmail.com>

On 12/30/2016 05:26 PM, Deborah Swanson wrote:
> I'm still wondering if these 4 lines can be collapsed to one or two
> lines.

If the logic is clearly expressed in the if blocks that you have, I
don't see why collapsing an if block into one or two lines would even be
desirable.  Making a clever one-liner out of something isn't always a
good thing.  In fact some programmers don't like to use the ternary
operator or conditional expressions, preferring to use explicit if block
logic.



From python at mrabarnett.plus.com  Fri Dec 30 21:47:45 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 31 Dec 2016 02:47:45 +0000
Subject: Cleaning up conditionals
In-Reply-To: <003801d262ff$931dec40$27b23dae@sambora>
References: <003801d262ff$931dec40$27b23dae@sambora>
Message-ID: <5081789f-c832-fc1d-3785-d9718257d173@mrabarnett.plus.com>

On 2016-12-31 00:48, Deborah Swanson wrote:
>> On 30/12/16 23:00, Deborah Swanson wrote:
>> > Oops, indentation was messed up when I copied it into the email.
>> > Should be this:
>> >
>> > 		if len(l1[st]) == 0:
>> >                 if len(l2[st]) > 0:
>> >                     l1[st] = l2[st]
>> >             elif len(l2[st]) == 0:
>> >                 if len(l1[st]) > 0:
>> >                     l2[st] = l1[st]
>>
>> That's even worse!
>>
>> Anyway, ignoring all that, if what you are trying to do is
>> just do some
>> action based on a set of fixed comparisons that are known at
>> the top of
>> the function (and do not mutate depending on the path through the
>> function), then you can just cache the comparisons and then
>> compare the
>> resulting set of boolean results (note that my examples are
>> NOT based on
>> your use-case, it's just pseudo-code which happens to use
>> your expressions):
>>
>> state = (len(l1[st]) == 0, len(l2[st]) > 0)
>> if state == (True, False):
>>    pass
>> elif state == (False, True):
>>    pass
>>
>> ... etc.
>>
>> If the len() comparisons are tri-state (i.e., in some cases
>> you want to
>> know if the length is <, ==, or > 0, depending on one of the other
>> comparisons) then you can do something like:
>>
>> def clamp(foo):
>>    return min(max(-1, foo), 1)
>>
>> state = (clamp(cmp(len(l1[st]), 0), cmp(len(l2[st]), 0))
>> if state == (0, -1):
>>    pass
>> elif state == (1, -1):
>>    pass
>>
>> ... etc.
>>
>> I'm not sure this makes it much more readable though - but if
>> you make
>> the RHS of those comparisons a symbolic name you might be getting
>> somewhere -
>>
>> ACTION1 = (0, -1)
>> ACTION2 = (1, -1)
>> if state == ACTION1:
>>    pass
>> elif state == ACTION2:
>>    pass
>>
>> Hope that helps, E.
>>
>
> Thanks Erik, I really like your method of capturing the state and then
> reusing it. I'm sure I can use it in other situations. In this case,
> since I now have it down to:
>
> if not len(l1[st]) and len(l2[st]):
> 		l1[st] = l2[st]
> elif not len(l2[st]) and len(l1[st]):
> 		l2[st] = l1[st]
>
[snip]

An empty list is considered falsey and a non-empty list is considered truey:

 >>> bool([])
False
 >>> bool([0])
True

You don't care what the list's length is, only whether it's 0 or not, so:

if not l1[st] and l2[st]:
     l1[st] = l2[st]
elif not l2[st] and l1[st]:
     l2[st] = l1[st]



From einstein1410 at gmail.com  Fri Dec 30 22:24:24 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 19:24:24 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.64.1483151164.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <58671740.7000908@stoneleaf.us>
 <mailman.64.1483151164.2395.python-list@python.org>
Message-ID: <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>

Ethan,
If you think I am wasting time, don't read my posts.
Why wasting your time in reading my post.

Are you having the answer of my question?
Till now no one is able to give answer, everybody just assuming something and answering. No answer is perfect that satisfy why it uses >>>. Except lan's answer.


Lan told that it is successor of ABC that's why it uses the >>> style.
Now I got my answer. But next question is Why it for ABC?
I know this is python group but it's reasonable doubt. If no doubts like this arises in your mind then you are just doing programming like Robot does and compiler converts the code in to machine language, it does not understand.
Same you are doing man.


Once again thanks a lot LAN for satisfiable answer.


From einstein1410 at gmail.com  Fri Dec 30 22:26:42 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 19:26:42 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.44.1483133004.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <148313078934.2259.7054529080896733101@mael>
 <mailman.44.1483133004.2395.python-list@python.org>
Message-ID: <68aaa874-abde-4dc9-83ad-265337705e41@googlegroups.com>

Thanks once again.
Its only you who gave me the satisfied answer.
Now it's my task to identify the reason why ABC used >>> ?


From einstein1410 at gmail.com  Fri Dec 30 22:28:56 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 19:28:56 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <58671740.7000908@stoneleaf.us>
 <mailman.64.1483151164.2395.python-list@python.org>
 <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>
Message-ID: <c2b17e8c-87c8-44ad-b835-0c3fefda980f@googlegroups.com>

Sorry everyone it's not LAN, its David Rogers.
Please accept my apology.


From einstein1410 at gmail.com  Fri Dec 30 22:39:14 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Fri, 30 Dec 2016 19:39:14 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <c2b17e8c-87c8-44ad-b835-0c3fefda980f@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <58671740.7000908@stoneleaf.us>
 <mailman.64.1483151164.2395.python-list@python.org>
 <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>
 <c2b17e8c-87c8-44ad-b835-0c3fefda980f@googlegroups.com>
Message-ID: <cb60f12d-46ca-4c9a-a7ac-caadebffdd57@googlegroups.com>

Thanks everyone. 
My all posted messages will be deleted within 1 hour.
I got my answer, its time to good bye.
If new questions arise then I will put it by creating new topic.
Good Bye all.
Once again thanks David Froger for your answer.


From rosuav at gmail.com  Fri Dec 30 22:44:10 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 31 Dec 2016 14:44:10 +1100
Subject: learning and experimenting python.
In-Reply-To: <CACL+1assa3vawJhHMgG1_FALLCXJEZKOHs2RmJu3AEjgk5ESEQ@mail.gmail.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <CACL+1assa3vawJhHMgG1_FALLCXJEZKOHs2RmJu3AEjgk5ESEQ@mail.gmail.com>
Message-ID: <CAPTjJmrsKO-OAG60WBS-1hBqzBsUy+JnBz5WX7AphTsT0SP0EA@mail.gmail.com>

On Sat, Dec 31, 2016 at 12:46 PM, eryk sun <eryksun at gmail.com> wrote:
> In modern Windows, cmd and PowerShell change the console title instead
> of the prompt. The title starts with "Administrator: " if the user is
> an administrator (i.e. the BUILTIN\Administrators group is present and
> enabled in the process token).

I'd much rather the prompt change. The title isn't always visible, and
even if it is, it isn't always obvious. When I'm ssh'ing around the
network, changing the title is a small convenience (since that means
the "running programs" list shows who I'm logged in as), but changing
the prompt is critical, because I see that right there as I start
typing. Fortunately, every Unix shell I've ever used lets me have both
- title AND prompt - so there need be no mistake.

(Of course, that still doesn't stop me fat-fingering and bringing
services down on the live system instead of the test...)

ChrisA


From python at mrabarnett.plus.com  Fri Dec 30 22:51:33 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 31 Dec 2016 03:51:33 +0000
Subject: Cleaning up conditionals
In-Reply-To: <20161231015930.GA73163@cskk.homeip.net>
References: <002401d262f2$f429bf30$27b23dae@sambora>
 <20161231015930.GA73163@cskk.homeip.net>
Message-ID: <69cbdab1-f425-04ee-1ae1-92c9ea55dc9b@mrabarnett.plus.com>

On 2016-12-31 01:59, Cameron Simpson wrote:
> On 30Dec2016 15:17, Deborah Swanson <python at deborahswanson.net> wrote:
>>> 	Ever consider using conjunctions?
>>>
>>> 	if len(l1[st]) and not len(l2[st]):
>>> 		#0 is considered a false -- no need to test for "==0"
>>> 		#non-0 is considered true -- no need to test for ">0"
>>> 		#copy l1 to l2
>>> 	elif not len(l1[st]) and len(l2[st]):
>>> 		#copy l2 to l1
>>> --
>>> 	Wulfraed                 Dennis Lee Bieber         AF6VN
>>>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/
>>
>>That's a neat shortcut, len(a) instead of len(a)!= 0.  Thanks!
>
> Also, for almost every python collection (lists, tuples, sets etc), python
> boolean logic tests __nonzero__, which works off len() by default.
>
[snip]

For Python 2, it tries __nonzero__, or __len__ if that fails.

For Python 3, it tries __bool__, or __len__ if that fails.



From python at deborahswanson.net  Fri Dec 30 22:52:22 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 19:52:22 -0800
Subject: Cleaning up conditionals
In-Reply-To: <20161231015930.GA73163@cskk.homeip.net>
Message-ID: <004201d26319$4b4745a0$27b23dae@sambora>

> On 30Dec2016 15:17, Deborah Swanson <python at deborahswanson.net> wrote:
> >> 	Ever consider using conjunctions?
> >>
> >> 	if len(l1[st]) and not len(l2[st]):
> >> 		#0 is considered a false -- no need to test for "==0"
> >> 		#non-0 is considered true -- no need to test for ">0"
> >> 		#copy l1 to l2
> >> 	elif not len(l1[st]) and len(l2[st]):
> >> 		#copy l2 to l1
> >> --
> >> 	Wulfraed                 Dennis Lee Bieber         AF6VN
> >>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/
> >
> >That's a neat shortcut, len(a) instead of len(a)!= 0.  Thanks!
> 
> Also, for almost every python collection (lists, tuples, sets 
> etc), python 
> boolean logic tests __nonzero__, which works off len() by default.
> 
> So:
> 
>   if a:
>     # a is not empty: len(a) > 0
>   else:
>     # a is empty: len(a) == 0
> 
> I find this far more readable, presuming the reader knows 
> that "empty" things 
> test as false. Of course, you need to ensure that any 
> "collection"-ish classes 
> you use or write have this property, but the builtin ones do.
> 
> Cheers,
> Cameron Simpson <cs at zip.com.au>

Another neat thing to know, and yes, that's much more readable. I've run
into trouble testing for empty (tests failed when they shouldn't have),
but I can't remember where I've had that problem, and since it happened
early in my learning python, chances are pretty good I screwed something
else up.  Thanks, I'll remember to try using it again  and see if I can
get it right.
 



From python at deborahswanson.net  Fri Dec 30 22:59:58 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 19:59:58 -0800
Subject: Cleaning up conditionals
In-Reply-To: <5081789f-c832-fc1d-3785-d9718257d173@mrabarnett.plus.com>
Message-ID: <004301d2631a$5aacf6b0$27b23dae@sambora>

> On 2016-12-31 00:48, Deborah Swanson wrote:
> >> On 30/12/16 23:00, Deborah Swanson wrote:
> >> > Oops, indentation was messed up when I copied it into the email. 
> >> > Should be this:
> >> >
> >> > 		if len(l1[st]) == 0:
> >> >                 if len(l2[st]) > 0:
> >> >                     l1[st] = l2[st]
> >> >             elif len(l2[st]) == 0:
> >> >                 if len(l1[st]) > 0:
> >> >                     l2[st] = l1[st]
> >>
> >> That's even worse!
> >>
> >> Anyway, ignoring all that, if what you are trying to do is just do 
> >> some action based on a set of fixed comparisons that are known at
> >> the top of
> >> the function (and do not mutate depending on the path through the
> >> function), then you can just cache the comparisons and then
> >> compare the
> >> resulting set of boolean results (note that my examples are
> >> NOT based on
> >> your use-case, it's just pseudo-code which happens to use
> >> your expressions):
> >>
> >> state = (len(l1[st]) == 0, len(l2[st]) > 0)
> >> if state == (True, False):
> >>    pass
> >> elif state == (False, True):
> >>    pass
> >>
> >> ... etc.
> >>
> >> If the len() comparisons are tri-state (i.e., in some 
> cases you want 
> >> to know if the length is <, ==, or > 0, depending on one 
> of the other
> >> comparisons) then you can do something like:
> >>
> >> def clamp(foo):
> >>    return min(max(-1, foo), 1)
> >>
> >> state = (clamp(cmp(len(l1[st]), 0), cmp(len(l2[st]), 0))
> >> if state == (0, -1):
> >>    pass
> >> elif state == (1, -1):
> >>    pass
> >>
> >> ... etc.
> >>
> >> I'm not sure this makes it much more readable though - but if you 
> >> make the RHS of those comparisons a symbolic name you might be 
> >> getting somewhere -
> >>
> >> ACTION1 = (0, -1)
> >> ACTION2 = (1, -1)
> >> if state == ACTION1:
> >>    pass
> >> elif state == ACTION2:
> >>    pass
> >>
> >> Hope that helps, E.
> >>
> >
> > Thanks Erik, I really like your method of capturing the 
> state and then 
> > reusing it. I'm sure I can use it in other situations. In 
> this case, 
> > since I now have it down to:
> >
> > if not len(l1[st]) and len(l2[st]):
> > 		l1[st] = l2[st]
> > elif not len(l2[st]) and len(l1[st]):
> > 		l2[st] = l1[st]
> >
> [snip]
> 
> An empty list is considered falsey and a non-empty list is 
> considered truey:
> 
>  >>> bool([])
> False
>  >>> bool([0])
> True
> 
> You don't care what the list's length is, only whether it's 0 
> or not, so:
> 
> if not l1[st] and l2[st]:
>      l1[st] = l2[st]
> elif not l2[st] and l1[st]:
>      l2[st] = l1[st]
> 

Similar, actually the same as Cameron suggested. I really need to
revisit testing for empty. I probably rejected it early on for some bad
reason (you don't understand everything that goes wrong when you're
learning).



From python at deborahswanson.net  Fri Dec 30 23:03:46 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Fri, 30 Dec 2016 20:03:46 -0800
Subject: Cleaning up conditionals
In-Reply-To: <60f00215-4a75-f017-4670-0cd794593dc6@gmail.com>
Message-ID: <004401d2631a$e2fe45a0$27b23dae@sambora>

Michael Torrie wrote:
> On 12/30/2016 05:26 PM, Deborah Swanson wrote:
> > I'm still wondering if these 4 lines can be collapsed to one or two 
> > lines.
> 
> If the logic is clearly expressed in the if blocks that you 
> have, I don't see why collapsing an if block into one or two 
> lines would even be desirable.  Making a clever one-liner out 
> of something isn't always a good thing.  In fact some 
> programmers don't like to use the ternary operator or 
> conditional expressions, preferring to use explicit if block logic.
> 

Maybe it isn't always a good thing, but learning the capabilities of
python is. Besides, if the concern is future maintenance, a lot would
depend on the proficiency of those expected to maintain the code.



From ian.g.kelly at gmail.com  Fri Dec 30 23:06:25 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 30 Dec 2016 22:06:25 -0600
Subject: learning and experimenting python.
In-Reply-To: <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
Message-ID: <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>

On Fri, Dec 30, 2016 at 6:05 PM,  <einstein1410 at gmail.com> wrote:
> But in his website, he recommended that post your questions here, he will answer it.
> But still as you told me I will send him an personal e-mail.

This is a good place for asking questions about Python, but you should
know that Guido himself doesn't usually post here.

> And if this is unimportant, as you thought, then you must not be the part of this group. This group is to share knowledge, nott for deciding what is trivia and what's not.

The information that you're asking for has no practical utility, hence
trivia. I wasn't passing judgment on your question; I was trying to
explain why it's unlikely that anybody would know the answer.

> My all posted messages will be deleted within 1 hour.

What makes you think that? This is an unmoderated Usenet group. The
only reason anything ever gets deleted from the archives is if it's
libelous spam. This is also distributed as a mailing list, and once
the email goes out, there's no unsending it.


From jsf80238 at gmail.com  Fri Dec 30 23:16:57 2016
From: jsf80238 at gmail.com (Jason Friedman)
Date: Fri, 30 Dec 2016 21:16:57 -0700
Subject: List comprehension
In-Reply-To: <5866fde5$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
 <mailman.40.1483126677.2395.python-list@python.org>
 <5866fde5$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CANy1k1jg9yHrj6-suFxwmYo4W3h_JyLRqtHzm78sQMHdpB_utw@mail.gmail.com>

>>>>> data = (
>> ... (1,2),
>> ... (3,4),
>> ... )
>>
>>>>> [x,y for a in data]
>>   File "<stdin>", line 1
>>     [x,y for a in data]
>>            ^
>> SyntaxError: invalid syntax
>>
>> I expected:
>> [(1, 2), (3, 4)]
>
>
> Why would you expect that? I would expect the global variables x and y, or
> if they don't exist, a NameError:

Thank you Steve (and everyone) for your answers.

In answer to Steve's question above, it's because I can do this:
>>> for x,y in data:
...     pass
...
>>>

But, anyway, I see my error now, and I'm good to go.


From tjreedy at udel.edu  Fri Dec 30 23:31:20 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 30 Dec 2016 23:31:20 -0500
Subject: List comprehension
In-Reply-To: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
References: <CANy1k1gxvuzMaZL-LFY=fuFv0wr_eHtb+MHZxoXrvskj_XbB+g@mail.gmail.com>
Message-ID: <o47cal$q1l$1@blaine.gmane.org>

On 12/30/2016 2:37 PM, Jason Friedman wrote:

> Now, this puzzles me:
>
>>>> [x,y for a in data]
>   File "<stdin>", line 1
>     [x,y for a in data]
>            ^
> SyntaxError: invalid syntax

I believe that python begins to parse this as
[x, (y for a in data)], a list of 2 items,
except that the required () are missing.
Notice that the ^ is under the r of 'for'.
"y for" is not a legal beginning of a list item.
Most tuples need ()s for proper grouping.



-- 
Terry Jan Reedy



From sagar.utlas at gmail.com  Fri Dec 30 23:53:37 2016
From: sagar.utlas at gmail.com (Sagar Utlas)
Date: Sat, 31 Dec 2016 10:23:37 +0530
Subject: Problem with running python 3.6.0 on a 32 bit windows 7 ultimate
 operating system.
In-Reply-To: <CADOGdD5mMvG64LNqXGdaJZ2GnZhAj7h3PTEOjR6vJWNb+TwDfg@mail.gmail.com>
References: <CADOGdD5mMvG64LNqXGdaJZ2GnZhAj7h3PTEOjR6vJWNb+TwDfg@mail.gmail.com>
Message-ID: <CADOGdD4LiSErrgZXLPx9iDSSrRTS9kv3NoiEwCNpF9J5Rdv6iQ@mail.gmail.com>

I am new to python, I've been using C++ as as a student till last 3 years.
To expand my knowledge, I installed Python 3.6.0 and when tried to open it,
a pop up window appeared saying- "The program can't start because
api-ms-win-crt-runtime-|1-1-0.dll is missing from your computer. try
reinstalling the program to fix this problem."
I tried repairing the software using the setup, but again it was the same.
What else can I do to run Python on my system.


From steve+python at pearwood.info  Sat Dec 31 00:03:36 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 16:03:36 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <o46beo$76k$1@dont-email.me>
 <96e184f9-7da9-4105-bdfa-7a0b6e7171bf@googlegroups.com>
Message-ID: <58673c29$0$1605$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 10:26 am, einstein1410 at gmail.com wrote:

> That's not the answer.
> If you don't have answer, please don't answer like this, because that will
> confuse others also.

What's not the answer? What's the question?


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From best_lay at yahoo.com  Sat Dec 31 00:22:19 2016
From: best_lay at yahoo.com (Wildman)
Date: Fri, 30 Dec 2016 23:22:19 -0600
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <adf2731a-5bf4-817f-6f32-a54f7de3c174@gmail.com>
 <mailman.63.1483151002.2395.python-list@python.org>
Message-ID: <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>

On Fri, 30 Dec 2016 19:23:17 -0700, Michael Torrie wrote:

> On 12/30/2016 07:05 PM, Wildman via Python-list wrote:
>> On Fri, 30 Dec 2016 23:39:43 +0000, Erik wrote:
>> 
>>> On 30/12/16 23:34, einstein1410 at gmail.com wrote:
>>>> You are also confusing me.
>>>> But there mustbe  some reason.
>>>> What happens if your student questions you like this.?
>>>> And may be those who questions like this will surely be the developer of its successor language.
>>>> Because out of thousands, only one may asks this, whom you all will consider fool, but he's the only genius
>>>
>>> Do not feed the troll.
>>>
>>> E.
>> 
>> Please explain how what I said is trolling.  Perhaps it was a little
>> snide but I tend to get that way when trying to explain the obvious.
> 
> Hmm. I thought he was referring to einstein1410...  It was his message
> he was replying to, not yours, and I took it as a request to the rest of
> us (sorry, Erik--couldn't resist posting).

I took it to mean that he was telling einstein1410 to not feed
the trolls.  If I am wrong then my apologies to Erik.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From steve+python at pearwood.info  Sat Dec 31 00:46:32 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 16:46:32 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <58671740.7000908@stoneleaf.us>
 <mailman.64.1483151164.2395.python-list@python.org>
 <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>
Message-ID: <5867463a$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 02:24 pm, einstein1410 at gmail.com wrote:

> Ethan,
> If you think I am wasting time, don't read my posts.
> Why wasting your time in reading my post.
> 
> Are you having the answer of my question?
> Till now no one is able to give answer, everybody just assuming something
> and answering. No answer is perfect that satisfy why it uses >>>. 

Guido picked >>> because he likes the look of it. When you create your own
programming language, you can pick any prompt you like. Different programs,
shells and languages use many different styles of prompts:

Python:         >>>
iPython:        In [1]:
Ruby (irb):     irb(main):001:0>
bash:           [steve at ando ~]$
bash (root):    [steve at ando ~]#
DOS:            >
rhino:          js>
lua:            >
julia:          julia>
xion:           >
mutt:           :
ed:             (no prompt)
zsh:            [steve at ando]~%


It is completely a matter of personal taste.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ian.g.kelly at gmail.com  Sat Dec 31 01:21:06 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sat, 31 Dec 2016 00:21:06 -0600
Subject: Problem with running python 3.6.0 on a 32 bit windows 7 ultimate
 operating system.
In-Reply-To: <CADOGdD4LiSErrgZXLPx9iDSSrRTS9kv3NoiEwCNpF9J5Rdv6iQ@mail.gmail.com>
References: <CADOGdD5mMvG64LNqXGdaJZ2GnZhAj7h3PTEOjR6vJWNb+TwDfg@mail.gmail.com>
 <CADOGdD4LiSErrgZXLPx9iDSSrRTS9kv3NoiEwCNpF9J5Rdv6iQ@mail.gmail.com>
Message-ID: <CALwzidkQ8TMeje03sKU5QZ5hxGRyznn5XtXZLQW5VDjEibZhZA@mail.gmail.com>

On Fri, Dec 30, 2016 at 10:53 PM, Sagar Utlas <sagar.utlas at gmail.com> wrote:
> I am new to python, I've been using C++ as as a student till last 3 years.
> To expand my knowledge, I installed Python 3.6.0 and when tried to open it,
> a pop up window appeared saying- "The program can't start because
> api-ms-win-crt-runtime-|1-1-0.dll is missing from your computer. try
> reinstalling the program to fix this problem."
> I tried repairing the software using the setup, but again it was the same.
> What else can I do to run Python on my system.

http://stackoverflow.com/questions/33265663/api-ms-win-crt-runtime-l1-1-0-dll-is-missing-when-opening-microsoft-office-file


From jussi.piitulainen at helsinki.fi  Sat Dec 31 01:54:12 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Sat, 31 Dec 2016 08:54:12 +0200
Subject: Cleaning up conditionals
References: <60f00215-4a75-f017-4670-0cd794593dc6@gmail.com>
 <004401d2631a$e2fe45a0$27b23dae@sambora>
 <mailman.72.1483156998.2395.python-list@python.org>
Message-ID: <lf5ful4io6j.fsf@ling.helsinki.fi>

"Deborah Swanson" <python at deborahswanson.net> writes:

> Michael Torrie wrote:
>> On 12/30/2016 05:26 PM, Deborah Swanson wrote:
>> > I'm still wondering if these 4 lines can be collapsed to one or two 
>> > lines.
>> 
>> If the logic is clearly expressed in the if blocks that you 
>> have, I don't see why collapsing an if block into one or two 
>> lines would even be desirable.  Making a clever one-liner out 
>> of something isn't always a good thing.  In fact some 
>> programmers don't like to use the ternary operator or 
>> conditional expressions, preferring to use explicit if block logic.
>> 
>
> Maybe it isn't always a good thing, but learning the capabilities of
> python is. Besides, if the concern is future maintenance, a lot would
> depend on the proficiency of those expected to maintain the code.

One line:

    l1[st], l2[st] = (l1[st] or l2[st]), (l2[st] or l1[st])

(The parentheses are redundant.)

Two lines:

    l1[st] = l1[st] or l2[st]
    l2[st] = l2[st] or l1[st]

Both of these store the same value back in the given field if it's
considered true (non-empty), else they store the corresponding value
from the other record in the hope that it would be considered true
(non-empty).


From einstein1410 at gmail.com  Sat Dec 31 03:25:41 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:25:41 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <5867463a$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <58671740.7000908@stoneleaf.us>
 <mailman.64.1483151164.2395.python-list@python.org>
 <9b19550d-02ed-41f8-8b97-9ff964d9e2e9@googlegroups.com>
 <5867463a$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <d3ebf87a-b895-4ac2-acf1-6c69541ae9e8@googlegroups.com>

Nice information.
Thanks a lot.

Krunalkumar Shah


From einstein1410 at gmail.com  Sat Dec 31 03:26:38 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:26:38 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <adf2731a-5bf4-817f-6f32-a54f7de3c174@gmail.com>
 <mailman.63.1483151002.2395.python-list@python.org>
 <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>
Message-ID: <7495e1ee-8092-4149-8e1f-6dfa889376bd@googlegroups.com>

Look I am not feeding trolls.
Please be aware.


From einstein1410 at gmail.com  Sat Dec 31 03:27:45 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:27:45 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.73.1483157229.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
 <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>
 <mailman.73.1483157229.2395.python-list@python.org>
Message-ID: <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>

It is moderatable. You can delete your all messages except topics.


From einstein1410 at gmail.com  Sat Dec 31 03:28:11 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:28:11 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.53.1483141372.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
Message-ID: <4afa4ad2-3b4c-42f6-8c5e-86f25ce5014e@googlegroups.com>

I am not feeding the troll.


From none at invalid.com  Sat Dec 31 03:30:09 2016
From: none at invalid.com (mm0fmf)
Date: Sat, 31 Dec 2016 08:30:09 +0000
Subject: learning and experimenting python.
In-Reply-To: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
Message-ID: <o47q7t$4jq$1@dont-email.me>

On 30/12/2016 17:50, einstein1410 at gmail.com wrote:
> Hello everyone,
> I am the new comer and learner of python.
> I have a doubt that when I type python and press enter it shows a prompt like
>>>>
> But why it is >>> ?
> Is there any special reason?
> Why it is not setted as @,& or any other special characters?
>

"I have a doubt" is not the correct way to ask this kind of question in 
English.

Doubt is used to mean lack of conviction or feeling of uncertainty so 
your question means "I feel uncertain that when I type Python the prompt 
will be >>>" and from the further responses this is not what you mean.

"I have a question about the prompt, why is it >>>" is far, far better.

HTH


From srikrishnamohan at gmail.com  Sat Dec 31 03:42:22 2016
From: srikrishnamohan at gmail.com (km)
Date: Sat, 31 Dec 2016 14:12:22 +0530
Subject: learning and experimenting python.
In-Reply-To: <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
 <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>
 <mailman.73.1483157229.2395.python-list@python.org>
 <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>
Message-ID: <CAPV1RACmBzW3MLD+EkG2BD7=gp98hCpbO=06Z8XK6=hqq77BNA@mail.gmail.com>

You are wasting our time instead of learning python.
On Dec 31, 2016 2:09 PM, <einstein1410 at gmail.com> wrote:

> It is moderatable. You can delete your all messages except topics.
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From einstein1410 at gmail.com  Sat Dec 31 03:47:13 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:47:13 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.78.1483173750.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
 <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>
 <mailman.73.1483157229.2395.python-list@python.org>
 <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>
 <CAPV1RACmBzW3MLD+EkG2BD7=gp98hCpbO=06Z8XK6=hqq77BNA@mail.gmail.com>
 <mailman.78.1483173750.2395.python-list@python.org>
Message-ID: <40408f6f-2b38-4eec-a6a6-d5742fcd38df@googlegroups.com>

Sorry for that.


From einstein1410 at gmail.com  Sat Dec 31 03:47:47 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 00:47:47 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <o47q7t$4jq$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
Message-ID: <8a619a1e-d2c2-4c00-8e81-906b5e34847b@googlegroups.com>

Thanks for suggesting me how to ask.
Thanks a lot.


From steve+python at pearwood.info  Sat Dec 31 04:34:53 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 20:34:53 +1100
Subject: Cleaning up conditionals
References: <CAMPXz=rb=D8pMBa7if4ZXMd9YN4KHtT+0y+mTy1WLDgEdhAH=A@mail.gmail.com>
 <003501d262fc$90a70030$27b23dae@sambora>
 <mailman.55.1483143930.2395.python-list@python.org>
Message-ID: <58677bbe$0$1591$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 11:26 am, Deborah Swanson wrote:

> As Mr. Bieber points out, what I had above greatly benefits from the use
> of conjunctions. It now reads:
> 
> if not len(l1[st]) and len(l2[st]):
>     l1[st] = l2[st]
> elif not len(l2[st]) and len(l1[st]):
>     l2[st] = l1[st]

Your code could do with more descriptive variable names, but keeping your
original names for the moment:

if not l1[st] and l2[st]:
    # first string is empty, second is not empty
    l1[st] = l2[st]
elif not l2[st] and l1[st]:
    # second string is empty, first is not empty
    l2[st] = l1[st]
    

The comments are, of course, redundant, and I probably wouldn't bother
putting them in my own code.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Dec 31 04:39:36 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 20:39:36 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
 <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>
 <mailman.73.1483157229.2395.python-list@python.org>
 <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>
Message-ID: <58677cda$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 07:27 pm, einstein1410 at gmail.com wrote:

> It is moderatable. You can delete your all messages except topics.

What *are* you talking about?

Do you realise that your messages go to at least one mailing list
(python-list at python.org), at least two news groups (comp.lang.python and
gmane.comp.python.general), Google Groups, and at least one other web
archive (ActiveState)?

Not to mention tens of thousands of people's private archives in their own
email or news client.

You are flooding this forum with dozens of short messages with no context.
This is not Twitter. Don't just hit send for every trivial thought that
enters your mind.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Dec 31 04:58:41 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 31 Dec 2016 20:58:41 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
Message-ID: <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>

On Sat, 31 Dec 2016 07:30 pm, mm0fmf wrote:

> On 30/12/2016 17:50, einstein1410 at gmail.com wrote:
>> Hello everyone,
>> I am the new comer and learner of python.
>> I have a doubt 
[...]

> "I have a doubt" is not the correct way to ask this kind of question in
> English.

But it is perfectly idiomatic Indian English, which is no less legitimate
than American English, Australian English, South African English and of
course the hundreds of dialects from the mother country, British English.

In Irish and Australian English, we're just as likely to say "I'm wondering
about..." rather than "I have a question".

This is an international forum, and we should make allowances for variant
forms of English and those whose grasp of the Queen's English is not
perfect.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From einstein1410 at gmail.com  Sat Dec 31 05:05:10 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 02:05:10 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <58677cda$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <5750c86d-336f-b727-1b61-9966ad49bc52@gmail.com>
 <mailman.41.1483127773.2395.python-list@python.org>
 <12e2948c-d26b-4080-878e-5c483e5920ec@googlegroups.com>
 <CALwzid=oBiQ+RtssZj_zVWBgBAh-dpCkePMF=1L1GGpzj8JnkA@mail.gmail.com>
 <mailman.54.1483142484.2395.python-list@python.org>
 <b6fa1a22-345c-4256-9654-a93997b7436e@googlegroups.com>
 <CALwzidnq0oxU2L9GWUSQJ8BXzg_JWKQHp-ktVUQA=jDqSTtC=Q@mail.gmail.com>
 <mailman.73.1483157229.2395.python-list@python.org>
 <08bad7e3-7cb2-439d-b5f6-7d678e9325f8@googlegroups.com>
 <58677cda$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <737eb5d2-cb4e-4a8d-a699-dccc90e87112@googlegroups.com>

I will
You can't stop me.


From einstein1410 at gmail.com  Sat Dec 31 05:05:37 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 02:05:37 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <d5a48cd6-fd0e-45eb-bac6-0bf7dc131250@googlegroups.com>

That's true.


From python at lucidity.plus.com  Sat Dec 31 06:57:12 2016
From: python at lucidity.plus.com (Erik)
Date: Sat, 31 Dec 2016 11:57:12 +0000
Subject: learning and experimenting python.
In-Reply-To: <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <adf2731a-5bf4-817f-6f32-a54f7de3c174@gmail.com>
 <mailman.63.1483151002.2395.python-list@python.org>
 <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>
Message-ID: <aea3d4e8-7e49-c991-c859-99a3152e34d9@lucidity.plus.com>

On 31/12/16 05:22, Wildman via Python-list wrote:
> On Fri, 30 Dec 2016 19:23:17 -0700, Michael Torrie wrote:
>
>> On 12/30/2016 07:05 PM, Wildman via Python-list wrote:
>>> On Fri, 30 Dec 2016 23:39:43 +0000, Erik wrote:
>>>> Do not feed the troll.
>>>>
>>>> E.
>>>
>>> Please explain how what I said is trolling.  Perhaps it was a little
>>> snide but I tend to get that way when trying to explain the obvious.
>>
>> Hmm. I thought he was referring to einstein1410...  It was his message
>> he was replying to, not yours, and I took it as a request to the rest of
>> us (sorry, Erik--couldn't resist posting).
>
> I took it to mean that he was telling einstein1410 to not feed
> the trolls.  If I am wrong then my apologies to Erik.

Yes, my message was to everyone, using a quote from 'einstein1410' that 
I thought demonstrated the point.

The vast majority of the messages in this thread are from 'einstein' nd 
are short, nonsensical retorts to others who are trying to be helpful, 
that just seem designed to get a further response and nothing more. 
Whether or not it's on purpose, he's just trolling. I may be wrong, but 
it just looks like the result of a bet to me - perhaps there's a similar 
thread on a Perl group somewhere and whoever gets the most replies wins.

E.


From none at invalid.com  Sat Dec 31 07:04:57 2016
From: none at invalid.com (mm0fmf)
Date: Sat, 31 Dec 2016 12:04:57 +0000
Subject: learning and experimenting python.
In-Reply-To: <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o486ql$ebi$1@dont-email.me>

On 31/12/2016 09:58, Steve D'Aprano wrote:
> On Sat, 31 Dec 2016 07:30 pm, mm0fmf wrote:
>
>> On 30/12/2016 17:50, einstein1410 at gmail.com wrote:
>>> Hello everyone,
>>> I am the new comer and learner of python.
>>> I have a doubt
> [...]
>
>> "I have a doubt" is not the correct way to ask this kind of question in
>> English.
>
> But it is perfectly idiomatic Indian English, which is no less legitimate
> than American English, Australian English, South African English and of
> course the hundreds of dialects from the mother country, British English.
>
> In Irish and Australian English, we're just as likely to say "I'm wondering
> about..." rather than "I have a question".
>
> This is an international forum, and we should make allowances for variant
> forms of English and those whose grasp of the Queen's English is not
> perfect.
>
>
>
>
Hence the suggestion to use a less idiomatic expression. I see this from 
my team of programmers in our Bangalore office regularly. Just about all 
of them were surprised when it was pointed out. Given emails are sent 
amongst that team and offices around the world, when they expect local 
team memebers to reply they now use "doubt" and when they'd like a 
global response they use "question". A brief skim of the mail lets you 
see if it needs dealing with immediately.

I've always found Australian English to be very similar to English 
English when written and only going "full down under" when spoken. 
Whereas American English is basically fewer words spelt differently. 
Though they do write road info upside down on the road surface, 
typically XING with PED underneath. I had to ask what XINGPED meant till 
my colleague driving said it was PEDXING, pedestrian crossing. Why write 
it the wrong way though?

I'm not sure if this thread isn't a most excellent troll however. I see 
it's almost 2017 where you are Steve... have a good one.


From marko at pacujo.net  Sat Dec 31 07:34:37 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Sat, 31 Dec 2016 14:34:37 +0200
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <o486ql$ebi$1@dont-email.me>
Message-ID: <87d1g8z38i.fsf@elektro.pacujo.net>

mm0fmf <none at invalid.com>:

> On 31/12/2016 09:58, Steve D'Aprano wrote:
>> On Sat, 31 Dec 2016 07:30 pm, mm0fmf wrote:
>>
>>> On 30/12/2016 17:50, einstein1410 at gmail.com wrote:
>>> "I have a doubt" is not the correct way to ask this kind of question in
>>> English.
>>
>> But it is perfectly idiomatic Indian English,
>
> Hence the suggestion to use a less idiomatic expression.

I have said it before that every English-speaker in the world should be
taught the international dialect of English, ie, Hollywood English. I
can understand virtually every foreign accent of English in the world,
but I have exceeding difficulties with the British accents.

BTW, it's interesting that with Brexit, only Ireland is left to hold the
candle for English in the EU.


Marko


From einstein1410 at gmail.com  Sat Dec 31 09:17:55 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 06:17:55 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <mailman.79.1483185435.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <CALwzidkOEKt2xh2qnqYYa=CKMtVKdE+mZtyO3Bst=+eLCYszjQ@mail.gmail.com>
 <mailman.39.1483123507.2395.python-list@python.org>
 <f1d8e362-3816-42ee-8931-ac148a6ebb51@googlegroups.com>
 <aKydnRtt3_BBKfvFnZ2dnUU7-X-dnZ2d@giganews.com>
 <2e056f55-6467-4faa-8583-4c1976c9513b@googlegroups.com>
 <b7faf02a-2bc1-cd54-7a18-e74c0b20ed82@lucidity.plus.com>
 <mailman.53.1483141372.2395.python-list@python.org>
 <MLSdnUJcZ8ocj_rFnZ2dnUU7-YWdnZ2d@giganews.com>
 <adf2731a-5bf4-817f-6f32-a54f7de3c174@gmail.com>
 <mailman.63.1483151002.2395.python-list@python.org>
 <VcednfjLf4AW3frFnZ2dnUU7-LmdnZ2d@giganews.com>
 <aea3d4e8-7e49-c991-c859-99a3152e34d9@lucidity.plus.com>
 <mailman.79.1483185435.2395.python-list@python.org>
Message-ID: <0dcb688e-0aa5-4a16-9573-b494f7415d61@googlegroups.com>

I warn you that I am not playing game here.
Please be cautioned. And if you think so, why post here?
Its seems that you are also interested in doing so.


From joel.goldstick at gmail.com  Sat Dec 31 10:34:20 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Sat, 31 Dec 2016 10:34:20 -0500
Subject: learning and experimenting python.
In-Reply-To: <1djf6c1artot2h5mdtthhbg1elk3bo36fh@4ax.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <o486ql$ebi$1@dont-email.me> <1djf6c1artot2h5mdtthhbg1elk3bo36fh@4ax.com>
Message-ID: <CAPM-O+wyFxDZdmyH3qQpkYdLCNbL7B+X6XJdAf4-Mp+Omw+Yfw@mail.gmail.com>

On Sat, Dec 31, 2016 at 10:27 AM, Dennis Lee Bieber
<wlfraed at ix.netcom.com> wrote:
> On Sat, 31 Dec 2016 12:04:57 +0000, mm0fmf <none at invalid.com> declaimed the
> following:
>
>>Though they do write road info upside down on the road surface,
>>typically XING with PED underneath. I had to ask what XINGPED meant till
>>my colleague driving said it was PEDXING, pedestrian crossing. Why write
>>it the wrong way though?
>
>         For those I've encountered with multiple lines of text, the lines are
> placed in the order one sees them.
>
>         Yes, in a helicopter they would be reversed, but from the view in a
> car, one sees the "PED" first, with the "XING" becoming visible as the
> "PED" begins to be blocked by the hood (or should I say "bonnet").

In the United States, ambulances often have their signage written
backwards so that it appears normal in a rear view mirror.  Do they do
that in other countries?


> --
>         Wulfraed                 Dennis Lee Bieber         AF6VN
>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/
>
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From none at invalid.com  Sat Dec 31 10:41:09 2016
From: none at invalid.com (mm0fmf)
Date: Sat, 31 Dec 2016 15:41:09 +0000
Subject: learning and experimenting python.
In-Reply-To: <mailman.81.1483198084.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <o486ql$ebi$1@dont-email.me> <1djf6c1artot2h5mdtthhbg1elk3bo36fh@4ax.com>
 <mailman.81.1483198084.2395.python-list@python.org>
Message-ID: <o48jg1$t7q$1@dont-email.me>

On 31/12/2016 15:27, Dennis Lee Bieber wrote:
> On Sat, 31 Dec 2016 12:04:57 +0000, mm0fmf <none at invalid.com> declaimed the
> following:
>
>> Though they do write road info upside down on the road surface,
>> typically XING with PED underneath. I had to ask what XINGPED meant till
>> my colleague driving said it was PEDXING, pedestrian crossing. Why write
>> it the wrong way though?
>
> 	For those I've encountered with multiple lines of text, the lines are
> placed in the order one sees them.
>
> 	Yes, in a helicopter they would be reversed, but from the view in a
> car, one sees the "PED" first, with the "XING" becoming visible as the
> "PED" begins to be blocked by the hood (or should I say "bonnet").
>
That could explain why there are so many crashes in the US, the drivers 
are looking at the road right in front of the hood/bonnet and not in the 
distance where you see XING and PED. I know some writing is written 
right to left and some is left to right and some is top to bottom but 
I've never come across bottom to top! ;-)


From ian.g.kelly at gmail.com  Sat Dec 31 10:46:13 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sat, 31 Dec 2016 09:46:13 -0600
Subject: learning and experimenting python.
In-Reply-To: <o47q7t$4jq$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
Message-ID: <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>

On Dec 31, 2016 1:48 AM, "mm0fmf" <none at invalid.com> wrote:

On 30/12/2016 17:50, einstein1410 at gmail.com wrote:

> Hello everyone,
> I am the new comer and learner of python.
> I have a doubt that when I type python and press enter it shows a prompt
> like
>
>>
>>>> But why it is >>> ?
> Is there any special reason?
> Why it is not setted as @,& or any other special characters?
>
>
"I have a doubt" is not the correct way to ask this kind of question in
English.

Doubt is used to mean lack of conviction or feeling of uncertainty so your
question means "I feel uncertain that when I type Python the prompt will be
>>>" and from the further responses this is not what you mean.


"Doubt" means "question" in Indian English. It's every bit as correct as
using "chips" to mean "French fries".


From ian.g.kelly at gmail.com  Sat Dec 31 10:51:29 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sat, 31 Dec 2016 09:51:29 -0600
Subject: learning and experimenting python.
In-Reply-To: <d5a48cd6-fd0e-45eb-bac6-0bf7dc131250@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <d5a48cd6-fd0e-45eb-bac6-0bf7dc131250@googlegroups.com>
Message-ID: <CALwzid=98ggNQiLNe0sVMbxN2wQ-zGecFvQf-DZ4_JRTjNZWuQ@mail.gmail.com>

On Dec 31, 2016 3:12 AM, <einstein1410 at gmail.com> wrote:

That's true.


Please include quoted context in your replies. I have no idea who or what
you're responding to.


From __peter__ at web.de  Sat Dec 31 11:24:35 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 31 Dec 2016 17:24:35 +0100
Subject: Cleaning up conditionals
References: <7a3c695c-7d20-4227-8e95-e5e3ebce359d@googlegroups.com>
 <000401d262e2$83f6b480$27b23dae@sambora>
Message-ID: <o48m3v$tk1$1@blaine.gmane.org>

Deborah Swanson wrote:

> Here I have a real mess, in my opinion:

[corrected code:]

>         if len(l1[st]) == 0:
>             if len(l2[st]) > 0:
>                 l1[st] = l2[st]
>         elif len(l2[st]) == 0:
>             if len(l1[st]) > 0:
>                 l2[st] = l1[st]

> Anybody know or see an easier (more pythonic) way to do this? I need to
> do it for four fields, and needless to say, that's a really long block
> of ugly code.

By "four fields", do you mean four values of st, or four pairs of l1, l2, or 
more elif-s with l3 and l4 -- or something else entirely?

Usually the most obvious way to avoid repetition is to write a function, and 
to make the best suggestion a bit more context is necessary.




From jussi.piitulainen at helsinki.fi  Sat Dec 31 11:29:48 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Sat, 31 Dec 2016 18:29:48 +0200
Subject: Cleaning up conditionals
References: <4jnd6cd7qk7uc49oldbttkr6jnnfqq3gvi@4ax.com>
 <002401d262f2$f429bf30$27b23dae@sambora>
 <mailman.51.1483139815.2395.python-list@python.org>
Message-ID: <lf57f6gkqo3.fsf@ling.helsinki.fi>

Deborah Swanson writes:

> Is it possible to use some version of the "a = expression1 if
> condition else expression2" syntax with an elif? And for expression1
> and expression2 to be single statements?  That's the kind of
> shortcutting I'd like to do, and it seems like python might be able to
> do something like this.

I missed this question when I read the thread earlier. The answer is
simply to make expression2 be another conditional expression. I tend to
write the whole chain in parentheses. This allows multi-line layouts
like the following alternatives:

a = ( first if len(first) > 0
      else second if len(second) > 0
      else make_stuff_up() )

a = ( first if len(first) > 0 else
      second if len(second) > 0 else
      make_stuff_up() )

Expression1 and expression2 cannot be statements. Python makes a formal
distinction between statements that have an effect and expressions that
have a value. All components of a conditional expression must be
expressions. A function call can behave either way but I think it good
style that the calls in expresions return values.


From none at invalid.com  Sat Dec 31 11:44:31 2016
From: none at invalid.com (mm0fmf)
Date: Sat, 31 Dec 2016 16:44:31 +0000
Subject: learning and experimenting python.
In-Reply-To: <mailman.83.1483199176.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>
 <mailman.83.1483199176.2395.python-list@python.org>
Message-ID: <o48n6r$brq$1@dont-email.me>

On 31/12/2016 15:46, Ian Kelly wrote:
> On Dec 31, 2016 1:48 AM, "mm0fmf" <none at invalid.com> wrote:
>
> On 30/12/2016 17:50, einstein1410 at gmail.com wrote:
>
>> Hello everyone,
>> I am the new comer and learner of python.
>> I have a doubt that when I type python and press enter it shows a prompt
>> like
>>
>>>
>>>>> But why it is >>> ?
>> Is there any special reason?
>> Why it is not setted as @,& or any other special characters?
>>
>>
> "I have a doubt" is not the correct way to ask this kind of question in
> English.
>
> Doubt is used to mean lack of conviction or feeling of uncertainty so your
> question means "I feel uncertain that when I type Python the prompt will be
>>>> " and from the further responses this is not what you mean.
>
>
> "Doubt" means "question" in Indian English. It's every bit as correct as
> using "chips" to mean "French fries".
>

I could agree with you regarding "chips" but then we'd both be wrong. ;-)

Other way round. Chips, meaning fried potato pieces as opposed to fried 
potato slices, were being made in the UK and Europe a long time before 
being named "French Fries" by Americans. "French Fries" are now made 
from reconsituted potato pulp extruded into shape as opposed to pieces 
of freshly cut potato. Although you do get US "French Fries" in the UK. 
UK chips are normally quite unlike "French Fries" served in US fast food 
style establishments. Typically on UK menus it will say "Fries" or 
"French Fries" if you are not getting chips.

For the record, the best chips I've had come from The Netherlands, 
Belgium and Yorkishire, England in that order.




From einstein1410 at gmail.com  Sat Dec 31 12:08:26 2016
From: einstein1410 at gmail.com (einstein1410 at gmail.com)
Date: Sat, 31 Dec 2016 09:08:26 -0800 (PST)
Subject: learning and experimenting python.
In-Reply-To: <o48n6r$brq$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>
 <mailman.83.1483199176.2395.python-list@python.org>
 <o48n6r$brq$1@dont-email.me>
Message-ID: <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>

We are not discussing here about English,
But for python. Don't divert. Other wise someone thinks that I am doing this for getting more posts.


From none at invalid.com  Sat Dec 31 12:24:44 2016
From: none at invalid.com (mm0fmf)
Date: Sat, 31 Dec 2016 17:24:44 +0000
Subject: OT: Re: learning and experimenting python.
In-Reply-To: <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>
 <mailman.83.1483199176.2395.python-list@python.org>
 <o48n6r$brq$1@dont-email.me>
 <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>
Message-ID: <o48pi8$kuf$1@dont-email.me>

On 31/12/2016 17:08, einstein1410 at gmail.com wrote:
> We are not discussing here about English,
> But for python. Don't divert. Other wise someone thinks that I am doing this for getting more posts.
>

Username einstein, asking bizarre questions, short responses with no 
included context. if this isn't trolling I'm a Dutchman.

Tot ziens!

p.s. Happy New Year to all on the nicest, most useful newsgroup still 
active on the net.





From joel.goldstick at gmail.com  Sat Dec 31 12:31:18 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Sat, 31 Dec 2016 12:31:18 -0500
Subject: learning and experimenting python.
In-Reply-To: <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>
 <mailman.83.1483199176.2395.python-list@python.org>
 <o48n6r$brq$1@dont-email.me>
 <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>
Message-ID: <CAPM-O+wqPk6v4RXFq+UXVqpq0Uv7Jcvt77QqKxzJTH5D-gXcTQ@mail.gmail.com>

On Sat, Dec 31, 2016 at 12:08 PM,  <einstein1410 at gmail.com> wrote:
> We are not discussing here about English,
> But for python. Don't divert. Other wise someone thinks that I am doing this for getting more posts.
> --
> https://mail.python.org/mailman/listinfo/python-list

If you stay around a while, einstein, you will see that the group will
wander off into (sometimes) interesting asides when the original
question seems to have been played out.  Its a form a creative
ignoring.

-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From python.list at tim.thechases.com  Sat Dec 31 15:41:21 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Sat, 31 Dec 2016 14:41:21 -0600
Subject: Cleaning up conditionals
In-Reply-To: <004301d2631a$5aacf6b0$27b23dae@sambora>
References: <5081789f-c832-fc1d-3785-d9718257d173@mrabarnett.plus.com>
 <004301d2631a$5aacf6b0$27b23dae@sambora>
Message-ID: <20161231144121.511b67b8@bigbox.christie.dr>

On 2016-12-30 19:59, Deborah Swanson wrote:
> Similar, actually the same as Cameron suggested. I really need to
> revisit testing for empty. I probably rejected it early on for some
> bad reason (you don't understand everything that goes wrong when
> you're learning).

If your data is anything like what I often get, it had spaces in it,
and

  x = " "
  if x:
    print("yep")
  else:
    print("nope")

prints "yep".  So when processing data like this, I prefer to take a
cleaning pass over everything to strip out leading/trailing
whitespace before attempting to use any logic on it.

-tkc




From python at deborahswanson.net  Sat Dec 31 15:57:13 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 12:57:13 -0800
Subject: Cleaning up conditionals
In-Reply-To: <lf57f6gkqo3.fsf@ling.helsinki.fi>
Message-ID: <001c01d263a8$76ca2320$27b23dae@sambora>

Jussi Piitulainen wrote:
> Sent: Saturday, December 31, 2016 8:30 AM
> Deborah Swanson writes:
> 
> > Is it possible to use some version of the "a = expression1 if 
> > condition else expression2" syntax with an elif? And for 
> expression1 
> > and expression2 to be single statements?  That's the kind of 
> > shortcutting I'd like to do, and it seems like python might 
> be able to 
> > do something like this.
> 
> I missed this question when I read the thread earlier. The 
> answer is simply to make expression2 be another conditional 
> expression. I tend to write the whole chain in parentheses. 
> This allows multi-line layouts like the following alternatives:
> 
> a = ( first if len(first) > 0
>       else second if len(second) > 0
>       else make_stuff_up() )
> 
> a = ( first if len(first) > 0 else
>       second if len(second) > 0 else
>       make_stuff_up() )
> 
> Expression1 and expression2 cannot be statements. Python 
> makes a formal distinction between statements that have an 
> effect and expressions that have a value. All components of a 
> conditional expression must be expressions. A function call 
> can behave either way but I think it good style that the 
> calls in expresions return values.

Your second alternative is exactly what I was thinking should be
possible in python, but I couldn't figure out how to do it. I'm
rewriting the little function I wrote to use this form. Thanks also for
the explanation of why expressions work in terniaries, but statements
don't, and what the difference between statements and expressions is.



From python at deborahswanson.net  Sat Dec 31 16:12:06 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 13:12:06 -0800
Subject: Cleaning up conditionals
In-Reply-To: <20161231144121.511b67b8@bigbox.christie.dr>
Message-ID: <001d01d263aa$8aeb43f0$27b23dae@sambora>

> From: Tim Chase
> Sent: Saturday, December 31, 2016 12:41 PM
> On 2016-12-30 19:59, Deborah Swanson wrote:
> > Similar, actually the same as Cameron suggested. I really need to 
> > revisit testing for empty. I probably rejected it early on for some 
> > bad reason (you don't understand everything that goes wrong when 
> > you're learning).
> 
> If your data is anything like what I often get, it had spaces 
> in it, and
> 
>   x = " "
>   if x:
>     print("yep")
>   else:
>     print("nope")
> 
> prints "yep".  So when processing data like this, I prefer to 
> take a cleaning pass over everything to strip out 
> leading/trailing whitespace before attempting to use any logic on it.
> 
> -tkc

Good point. Like I said earlier, I don't remember the specific contexts
that testing for empty failed for me, but blank spaces in data that I
didn't realize would be there (or look for it) is a strong possibility.
In any event, I should use tests for empty more, and see if and why it
fails. Obviously, experienced python coders successfully use it all the
time. 



From python at deborahswanson.net  Sat Dec 31 17:03:05 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 14:03:05 -0800
Subject: Cleaning up conditionals
In-Reply-To: <lf57f6gkqo3.fsf@ling.helsinki.fi>
Message-ID: <002001d263b1$a9e3de50$27b23dae@sambora>

Jussi Piitulainen wrote:
> Sent: Saturday, December 31, 2016 8:30 AM
> Deborah Swanson writes:
> 
> > Is it possible to use some version of the "a = expression1 if 
> > condition else expression2" syntax with an elif? And for 
> expression1 
> > and expression2 to be single statements?  That's the kind of 
> > shortcutting I'd like to do, and it seems like python might 
> be able to 
> > do something like this.
> 
> I missed this question when I read the thread earlier. The 
> answer is simply to make expression2 be another conditional 
> expression. I tend to write the whole chain in parentheses. 
> This allows multi-line layouts like the following alternatives:
> 
> a = ( first if len(first) > 0
>       else second if len(second) > 0
>       else make_stuff_up() )
> 
> a = ( first if len(first) > 0 else
>       second if len(second) > 0 else
>       make_stuff_up() )
> 
> Expression1 and expression2 cannot be statements. Python 
> makes a formal distinction between statements that have an 
> effect and expressions that have a value. All components of a 
> conditional expression must be expressions. A function call 
> can behave either way but I think it good style that the 
> calls in expresions return values.

While I'm sure these terniaries will be useful for future problems, I
couldn't make the second one work for my current problem. I got as far
as:

a = l1[v] if len(l1[v] > 0 else 
    l2[v] if len(l2[v] > 0 else

And didn't finish it because I couldn't see what a should be. I want it
to be 
l2[v] if the first clause is true, and l1[v] if the second. If I was
computing a value, this would work beautifully, but I don't see how it
can if I'm choosing a list element to assign to. Maybe I just can't see
it.



From rosuav at gmail.com  Sat Dec 31 17:18:30 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Jan 2017 09:18:30 +1100
Subject: Cleaning up conditionals
In-Reply-To: <002001d263b1$a9e3de50$27b23dae@sambora>
References: <lf57f6gkqo3.fsf@ling.helsinki.fi>
 <002001d263b1$a9e3de50$27b23dae@sambora>
Message-ID: <CAPTjJmrm7qT-vN6KmfRT_3+1kajCPtuouVthVdu7rcV9w3xfeQ@mail.gmail.com>

On Sun, Jan 1, 2017 at 9:03 AM, Deborah Swanson
<python at deborahswanson.net> wrote:
> And didn't finish it because I couldn't see what a should be. I want it
> to be
> l2[v] if the first clause is true, and l1[v] if the second. If I was
> computing a value, this would work beautifully, but I don't see how it
> can if I'm choosing a list element to assign to. Maybe I just can't see
> it.

It's possible to select either l1 or l2 using an expression, and then
subscript that with [v]. However, this does not usually make for
readable code, so I don't recommend it.

(l1 if whatever else l2)[v] = new_value

ChrisA


From python at deborahswanson.net  Sat Dec 31 17:35:46 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 14:35:46 -0800
Subject: Cleaning up conditionals
In-Reply-To: <o48m3v$tk1$1@blaine.gmane.org>
Message-ID: <002101d263b6$3ae79960$27b23dae@sambora>

Peter Otten wrote:
> Deborah Swanson wrote:
> 
> > Here I have a real mess, in my opinion:
> 
> [corrected code:]
> 
> >         if len(l1[st]) == 0:
> >             if len(l2[st]) > 0:
> >                 l1[st] = l2[st]
> >         elif len(l2[st]) == 0:
> >             if len(l1[st]) > 0:
> >                 l2[st] = l1[st]
> 
> > Anybody know or see an easier (more pythonic) way to do 
> this? I need 
> > to do it for four fields, and needless to say, that's a really long 
> > block of ugly code.
> 
> By "four fields", do you mean four values of st, or four 
> pairs of l1, l2, or 
> more elif-s with l3 and l4 -- or something else entirely?
> 
> Usually the most obvious way to avoid repetition is to write 
> a function, and 
> to make the best suggestion a bit more context is necessary.
> 

I did write a function for this, and welcome any suggestions for
improvement.

The context is comparing 2 adjacent rows of data (in a list of real
estate listings sorted by their webpage titles and dates) with the
assumption that if the webpage titles are the same, they're listings for
the same property. This assumption is occasionally bad, but in far less
than one per 1000 unique listings. I'd rather just hand edit the data in
those cases so one webpage title is slightly different, than writing and
executing all the code needed to find and handle these corner cases.
Maybe that will be a future refinement, but right now I don't really
need it.

Once two rows of listing data have been identified as different dates
for the same property, there are 4 fields that will be identical for
both rows. There can be up to 10 (or even more) listings identical
except for the date, but typically I'm just adding a new one and want to
copy the field data from its previous siblings, so the copying is just
from the last listing to the new one.

Here's the function I have so far:

def comprows(l1,l2,st,ki,no):
    ret = ''
    labels = {st: 'st/co', ki: 'kind', no: 'notes'}
    for v in (st,ki,no):
        if len(l1[v]) == 0 and len(l2[v]) != 0:
            l1[v] = l2[v]
        elif len(l2[v]) == 0 and len(l1[v]) != 0:
            l2[v] = l1[v]
        elif l1[v] != l2[v]:
            ret += ", " + labels[v] + " diff" if len(ret) > 0 else 
            labels[v] + " diff"
    return ret

The 4th field is a special case and easily dispatched in one line of
code before this function is called for the other 3.

l1 and l2 are the 2 adjacent rows of listing data, with st,ki,no holding
codes for state/county, kind (of property) and notes. I want the
checking and copying to go both ways because sometimes I'm backfilling
old listings that I didn't pick up in my nightly copies on their given
dates, but came across them later.

ret is returned to a field with details to look at when I save the list
to csv and open it in Excel. The noted diffs will need to be reconciled.

I tried to use Jussi Piitulainen's suggestion to chain the conditionals,
but just couldn't make it work for choosing list elements to assign to,
although the approach is perfect if you're computing a value.

Hope this is enough context... ;)
D  



From steve+python at pearwood.info  Sat Dec 31 18:53:00 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 01 Jan 2017 10:53:00 +1100
Subject: learning and experimenting python.
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <o486ql$ebi$1@dont-email.me> <1djf6c1artot2h5mdtthhbg1elk3bo36fh@4ax.com>
 <mailman.81.1483198084.2395.python-list@python.org>
 <o48jg1$t7q$1@dont-email.me> <4o1g6c9qej7nol95e3pbcranqm85rjvik7@4ax.com>
 <mailman.88.1483212859.2395.python-list@python.org>
Message-ID: <586844de$0$1590$c3e8da3$5496439d@news.astraweb.com>

On Sun, 1 Jan 2017 06:34 am, Dennis Lee Bieber wrote:

> On Sat, 31 Dec 2016 15:41:09 +0000, mm0fmf <none at invalid.com> declaimed
> the following:
> 
> 
>>That could explain why there are so many crashes in the US, the drivers
>>are looking at the road right in front of the hood/bonnet 

This fortunately is not a problem in Australia, where the drivers' eyes are
firmly fixed at all times to the iPhone they are holding just below the
steering wheel.



>>and not in the 
>>distance where you see XING and PED. I know some writing is written
> 
> For the most part, if those markings are in the distance, the effects
> of foreshortening makes them illegible. You have to be in a few car
> lengths before you can read them (and the upper line may still be obscured
> by the vehicle in front of you -- so again the odds are that you will see
> the lower "PED" first, then see the upper "XING").

I wonder whether there is any objective evidence for the usefulness of text
written on the road surface, as opposed to road markings (the old-fashioned
zebra crossing, stripes) and/or road signs.

In Melbourne, at least in the areas I frequent, we don't have many examples
of text written on the road surface myself. But when I do see them, I find
them more distracting than helpful.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From python at mrabarnett.plus.com  Sat Dec 31 19:06:02 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 1 Jan 2017 00:06:02 +0000
Subject: [SPAM] RE: Cleaning up conditionals
In-Reply-To: <002101d263b6$3ae79960$27b23dae@sambora>
References: <002101d263b6$3ae79960$27b23dae@sambora>
Message-ID: <1cf7c684-74a7-8f1f-d97d-98d802a8124a@mrabarnett.plus.com>

On 2016-12-31 22:35, Deborah Swanson wrote:
> Peter Otten wrote:
>> Deborah Swanson wrote:
>>
>> > Here I have a real mess, in my opinion:
>>
>> [corrected code:]
>>
>> >         if len(l1[st]) == 0:
>> >             if len(l2[st]) > 0:
>> >                 l1[st] = l2[st]
>> >         elif len(l2[st]) == 0:
>> >             if len(l1[st]) > 0:
>> >                 l2[st] = l1[st]
>>
>> > Anybody know or see an easier (more pythonic) way to do
>> this? I need
>> > to do it for four fields, and needless to say, that's a really long
>> > block of ugly code.
>>
>> By "four fields", do you mean four values of st, or four
>> pairs of l1, l2, or
>> more elif-s with l3 and l4 -- or something else entirely?
>>
>> Usually the most obvious way to avoid repetition is to write
>> a function, and
>> to make the best suggestion a bit more context is necessary.
>>
>
> I did write a function for this, and welcome any suggestions for
> improvement.
>
> The context is comparing 2 adjacent rows of data (in a list of real
> estate listings sorted by their webpage titles and dates) with the
> assumption that if the webpage titles are the same, they're listings for
> the same property. This assumption is occasionally bad, but in far less
> than one per 1000 unique listings. I'd rather just hand edit the data in
> those cases so one webpage title is slightly different, than writing and
> executing all the code needed to find and handle these corner cases.
> Maybe that will be a future refinement, but right now I don't really
> need it.
>
> Once two rows of listing data have been identified as different dates
> for the same property, there are 4 fields that will be identical for
> both rows. There can be up to 10 (or even more) listings identical
> except for the date, but typically I'm just adding a new one and want to
> copy the field data from its previous siblings, so the copying is just
> from the last listing to the new one.
>
> Here's the function I have so far:
>
> def comprows(l1,l2,st,ki,no):
>     ret = ''
>     labels = {st: 'st/co', ki: 'kind', no: 'notes'}
>     for v in (st,ki,no):
>         if len(l1[v]) == 0 and len(l2[v]) != 0:
>             l1[v] = l2[v]
>         elif len(l2[v]) == 0 and len(l1[v]) != 0:
>             l2[v] = l1[v]
>         elif l1[v] != l2[v]:
>             ret += ", " + labels[v] + " diff" if len(ret) > 0 else
>             labels[v] + " diff"
>     return ret
>
> The 4th field is a special case and easily dispatched in one line of
> code before this function is called for the other 3.
>
> l1 and l2 are the 2 adjacent rows of listing data, with st,ki,no holding
> codes for state/county, kind (of property) and notes. I want the
> checking and copying to go both ways because sometimes I'm backfilling
> old listings that I didn't pick up in my nightly copies on their given
> dates, but came across them later.
>
> ret is returned to a field with details to look at when I save the list
> to csv and open it in Excel. The noted diffs will need to be reconciled.
>
> I tried to use Jussi Piitulainen's suggestion to chain the conditionals,
> but just couldn't make it work for choosing list elements to assign to,
> although the approach is perfect if you're computing a value.
>
> Hope this is enough context... ;)
> D
>
Here's a slightly different way of doing it:

def comprows(l1, l2, st, ki, no):
     ret = ''
     labels = {st: 'st/co', ki: 'kind', no: 'notes'}
     for v in (st, ki, no):
         t = list({l1[v], l2[v]} - {''})
         if len(t) == 1:
             l1[v] = l2[v] = t[0]
         elif len(t) == 2:
             ret += ", " + labels[v] + " diff"
     return ret[2 : ]


And here's a summary of what it does:

If l1[v] == l2[v], then {l1[v], l2[v]} will contain 1 string, otherwise 
it'll contain 2 strings. Then remove any empty string.

If the set now contains 1 string, then either they were the same, or one 
of them was empty; in either case, just make them the same.

On the other hand, if the set contains 2 strings, then report that they 
were different.



From greg.ewing at canterbury.ac.nz  Sat Dec 31 19:46:10 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Sun, 01 Jan 2017 13:46:10 +1300
Subject: learning and experimenting python.
In-Reply-To: <mailman.82.1483198464.2395.python-list@python.org>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <58678152$0$1622$c3e8da3$5496439d@news.astraweb.com>
 <o486ql$ebi$1@dont-email.me> <1djf6c1artot2h5mdtthhbg1elk3bo36fh@4ax.com>
 <CAPM-O+wyFxDZdmyH3qQpkYdLCNbL7B+X6XJdAf4-Mp+Omw+Yfw@mail.gmail.com>
 <mailman.82.1483198464.2395.python-list@python.org>
Message-ID: <ecr1qkFg8fbU1@mid.individual.net>

Joel Goldstick wrote:
> In the United States, ambulances often have their signage written
> backwards so that it appears normal in a rear view mirror.  Do they do
> that in other countries?

In NZ our fire engines have FIRE written both forwards and
backwards on the front, so it's readable either way. Not
sure about ambulances offhand, I'll check next time I see
one.

We also have CROSSING RAILWAY and BRIDGE LANE ONE signs
written on the road surface. :-)

-- 
Greg


From greg.ewing at canterbury.ac.nz  Sat Dec 31 19:49:51 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Sun, 01 Jan 2017 13:49:51 +1300
Subject: OT: Re: learning and experimenting python.
In-Reply-To: <o48pi8$kuf$1@dont-email.me>
References: <6886d439-751c-42d2-adf2-fbf494221d48@googlegroups.com>
 <o47q7t$4jq$1@dont-email.me>
 <CALwzidmJpvCxLMi59x=ZaP65idCa8ueR93oxJRg6YERJsSh6ew@mail.gmail.com>
 <mailman.83.1483199176.2395.python-list@python.org>
 <o48n6r$brq$1@dont-email.me>
 <1657c9c6-66af-4340-8e23-04b09da08d1e@googlegroups.com>
 <o48pi8$kuf$1@dont-email.me>
Message-ID: <ecr21hFg9qaU1@mid.individual.net>

mm0fmf wrote:
> Username einstein, asking bizarre questions, short responses with no 
> included context. if this isn't trolling I'm a Dutchman.

I don't think he's trolling, I think he's using some kind of
forum interface (Google Groups?) that displays whole threads
together, making quoting less necessary.

He seems to be under the impression that he will be able to
delete all his trivial posts later, and no harm will have
been done.

To einstein1410: You will *not* be able to delete those posts,
because they are being gatewayed to various archived mailing
lists and news groups. You may be able to make them disppear
from your view of the group, but they will remain in those
archives.

-- 
Greg


From python at deborahswanson.net  Sat Dec 31 20:42:19 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 17:42:19 -0800
Subject: learning and experimenting python.
In-Reply-To: <aea3d4e8-7e49-c991-c859-99a3152e34d9@lucidity.plus.com>
Message-ID: <003b01d263d0$4a633a10$27b23dae@sambora>

Erik wrote:
> Yes, my message was to everyone, using a quote from 
> 'einstein1410' that 
> I thought demonstrated the point.
> 
> The vast majority of the messages in this thread are from 
> 'einstein' nd 
> are short, nonsensical retorts to others who are trying to be 
> helpful, 
> that just seem designed to get a further response and nothing more. 
> Whether or not it's on purpose, he's just trolling. <snip>
> 
> E.

I quite agree, though it's hard to say what einstein's problem is.
He/She seems like a child to me, and that might explain why he/she
doesn't recognize the differences between internet services: Twitter,
mailing lists - moderated and not, newsgroups, etc - they're all "on the
internet". He/She doesn't seem to know where he is or what behavior is
appropriate to different services or even that there are differences,
like someone with little contact with these environments, and I'd guess
that Twitter is his/her model for how to do it. It's also likely that a
child would have fantasies about the meaning and importance of
unfamiliar things, and perhaps assume that some parent-like entity would
understand, explain and clean up after him/her.



From __peter__ at web.de  Sat Dec 31 21:16:00 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 01 Jan 2017 03:16 +0100
Subject: Cleaning up conditionals
References: <o48m3v$tk1$1@blaine.gmane.org>
 <002101d263b6$3ae79960$27b23dae@sambora>
Message-ID: <o49oou$hha$1@blaine.gmane.org>

Deborah Swanson wrote:

> Peter Otten wrote:
>> Deborah Swanson wrote:
>> 
>> > Here I have a real mess, in my opinion:
>> 
>> [corrected code:]
>> 
>> >         if len(l1[st]) == 0:
>> >             if len(l2[st]) > 0:
>> >                 l1[st] = l2[st]
>> >         elif len(l2[st]) == 0:
>> >             if len(l1[st]) > 0:
>> >                 l2[st] = l1[st]
>> 
>> > Anybody know or see an easier (more pythonic) way to do
>> this? I need
>> > to do it for four fields, and needless to say, that's a really long
>> > block of ugly code.
>> 
>> By "four fields", do you mean four values of st, or four
>> pairs of l1, l2, or
>> more elif-s with l3 and l4 -- or something else entirely?
>> 
>> Usually the most obvious way to avoid repetition is to write
>> a function, and
>> to make the best suggestion a bit more context is necessary.
>> 
> 
> I did write a function for this, and welcome any suggestions for
> improvement.
> 
> The context is comparing 2 adjacent rows of data (in a list of real
> estate listings sorted by their webpage titles and dates) with the
> assumption that if the webpage titles are the same, they're listings for
> the same property. This assumption is occasionally bad, but in far less
> than one per 1000 unique listings. I'd rather just hand edit the data in
> those cases so one webpage title is slightly different, than writing and
> executing all the code needed to find and handle these corner cases.
> Maybe that will be a future refinement, but right now I don't really
> need it.
> 
> Once two rows of listing data have been identified as different dates
> for the same property, there are 4 fields that will be identical for
> both rows. There can be up to 10 (or even more) listings identical
> except for the date, but typically I'm just adding a new one and want to
> copy the field data from its previous siblings, so the copying is just
> from the last listing to the new one.
> 
> Here's the function I have so far:
> 
> def comprows(l1,l2,st,ki,no):
>     ret = ''
>     labels = {st: 'st/co', ki: 'kind', no: 'notes'}
>     for v in (st,ki,no):
>         if len(l1[v]) == 0 and len(l2[v]) != 0:
>             l1[v] = l2[v]
>         elif len(l2[v]) == 0 and len(l1[v]) != 0:
>             l2[v] = l1[v]
>         elif l1[v] != l2[v]:
>             ret += ", " + labels[v] + " diff" if len(ret) > 0 else
>             labels[v] + " diff"
>     return ret
> 
> The 4th field is a special case and easily dispatched in one line of
> code before this function is called for the other 3.
> 
> l1 and l2 are the 2 adjacent rows of listing data, with st,ki,no holding
> codes for state/county, kind (of property) and notes. I want the
> checking and copying to go both ways because sometimes I'm backfilling
> old listings that I didn't pick up in my nightly copies on their given
> dates, but came across them later.
> 
> ret is returned to a field with details to look at when I save the list
> to csv and open it in Excel. The noted diffs will need to be reconciled.
> 
> I tried to use Jussi Piitulainen's suggestion to chain the conditionals,
> but just couldn't make it work for choosing list elements to assign to,
> although the approach is perfect if you're computing a value.
> 
> Hope this is enough context... ;)

At least the code into which I translate your description differs from the 
suggestions you have got so far. The main differences:

- Look at the whole group, not just two lines
- If there is more than one non-empty value in the group don't change any
  value.


from collections import defaultdict

def get_title(row):
    return row[...]

def complete(group, label):
    """For every row in the group set row[label] to a non-empty value
    if there is exactly one such value.

    Returns True if values can be set consistently.
    group is supposed to be a list of dicts.

    >>> def c(g):
    ...     gg = [{"whatever": value} for value in g]
    ...     if not complete(gg, "whatever"):
    ...         print("fixme", end=" ")
    ...     return [row["whatever"] for row in gg]
    >>> c(["", "a", ""])
    ['a', 'a', 'a']
    >>> c(["", "a", "a"])
    ['a', 'a', 'a']
    >>> c(["", "a", "b"])
    fixme ['', 'a', 'b']
    >>> c(["a"])
    ['a']
    >>> c([''])
    fixme ['']
    """
    values = {row[label] for row in group}
    has_empty = not min(values, key=len)
    if len(values) - has_empty != 1:
        # no value or multiple values; manual intervention needed
        return False
    elif has_empty:
        for row in group:
            row[label] = max(values, key=len)
    return True

if __name__ == "__main__":
    # read rows
    rows = ...

    # group rows by title
    groups = collections.defaultdict(list)
    for row in rows:
        groups[get_title(row)].append(row)

    LABELS = ['st/co', 'kind', 'notes']

    # add missing values
    for group in groups.values():
        for label in LABELS:
            complete(group, label)

    # write rows
    ...




From python at deborahswanson.net  Sat Dec 31 22:58:36 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 19:58:36 -0800
Subject: Cleaning up conditionals
In-Reply-To: <CAPTjJmrm7qT-vN6KmfRT_3+1kajCPtuouVthVdu7rcV9w3xfeQ@mail.gmail.com>
Message-ID: <004c01d263e3$5420fde0$27b23dae@sambora>

Chris Angelico wrote:
> Sent: Saturday, December 31, 2016 2:19 PM
> 
> On Sun, Jan 1, 2017 at 9:03 AM, Deborah Swanson 
> <python at deborahswanson.net> wrote:
> > And didn't finish it because I couldn't see what a should 
> be. I want 
> > it to be l2[v] if the first clause is true, and l1[v] if 
> the second. 
> > If I was computing a value, this would work beautifully, 
> but I don't 
> > see how it can if I'm choosing a list element to assign to. Maybe I 
> > just can't see it.
> 
> It's possible to select either l1 or l2 using an expression, 
> and then subscript that with [v]. However, this does not 
> usually make for readable code, so I don't recommend it.
> 
> (l1 if whatever else l2)[v] = new_value
> 
> ChrisA

I'm not sure I understand what you did here, at least not well enough to
try it.

What conditional can I do between the 2 rows of listings (the list names
l1 and l2) that will give me which row has the value to copy from and
which one is empty? I don't see how that can happen if you don't give
the subscript [v] to each of l1 and l2, at a minimum. Unless python will
distribute the [v] inside the preceding conditional?  Amazing, if true,
but then we still need what the conditional is.

And what is new_value? It could be either l1[v] or l2[v], depending on
which one is not empty, and I don't see how the answer magically pops
into it.  Not saying that it doesn't, but what should I call new_value
in the real estate listings example I want to use it in?  There are no
new values in this situation, only values that need to be copied into
their empty counterparts in the other row.  (It may or may not help to
read the synopsis of what I'm doing that I wrote up in my last post to
Peter Otten.)

No, it's not terribly readable, but I'm curious whether this actually
works.

D.




From python at deborahswanson.net  Sat Dec 31 23:17:15 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Sat, 31 Dec 2016 20:17:15 -0800
Subject: Cleaning up conditionals
In-Reply-To: <aatg6clp40g9qclbo29rdufc6tb0pjkgd9@4ax.com>
Message-ID: <004f01d263e5$ef64b1a0$27b23dae@sambora>

Dennis Lee Bieber
> Sent: Saturday, December 31, 2016 7:29 PM
> 
> On Sat, 31 Dec 2016 14:35:46 -0800, "Deborah Swanson" 
> <python at deborahswanson.net> declaimed the following:
> 
> >Here's the function I have so far:
> >
> >def comprows(l1,l2,st,ki,no):
> >    ret = ''
> >    labels = {st: 'st/co', ki: 'kind', no: 'notes'}
> >    for v in (st,ki,no):
> >        if len(l1[v]) == 0 and len(l2[v]) != 0:
> >            l1[v] = l2[v]
> >        elif len(l2[v]) == 0 and len(l1[v]) != 0:
> >            l2[v] = l1[v]
> >        elif l1[v] != l2[v]:
> >            ret += ", " + labels[v] + " diff" if len(ret) > 0 else 
> >            labels[v] + " diff"
> >    return ret
> >
> 
> 	Now, out of all that, which is the most likely one to 
> be activated? Granted, it's too early to be considering 
> optimizations, but... If you expect the common situation to 
> be the "l1 != l2", the order of your tests means you do two 
> failing tests before getting to the most likely version.
> -- 
> 	Wulfraed                 Dennis Lee Bieber         AF6VN
>     wlfraed at ix.netcom.com    HTTP://wlfraed.home.netcom.com/

I see what you're saying, but actually it's fairly rare that l1[v] and
l2[v] are different. I just need to catch it and look at it when they
are. It usually means that either the spelling/typing in the listing was
wrong and my code didn't fix it, or in rarer cases (now), my code is
buggy. I probably won't ever be able to dispense with this test since
people are always typoing. They don't always copy from their previous
listings and sometimes it looks like they're trying to retype it from
memory, and messing up.
D.