On Fri, Nov 15, 2013 at 9:10 PM, Tim Peters wrote:

...

_DI4Y = _days_before_year(5) # A 4-year cycle has an extra leap day over what we'd get from pasting # together 4 single years. assert _DI4Y == 4 * 365 + 1

To me, the constant should be directly set to its known value. _DI4Y = 4*365 + 1. The function should then be tested in test_datetime. self.assertEqual(dt._days_before_year(5), dt._DI4Y)

I think making that change would be pointless code churn. Harmful, even. As the guy who happened to have written that code ;-), I think it's valuable to have the _code_ (not off buried in some monstrously tedious test file) explain what the comments there do explain, and verify with the assert. If anyone needs to muck with the implementation of datetime, it's crucial they understand what DI4Y _means_, and that it's identical to _days_before_year(5). Its actual value (4*365 + 1) isn't really interesting. Defining _DI4Y _as_ _days_before_year(5) captures its _meaning_.

Interestingly, the corresponding C code is closer to what Terry suggested: /* Number of days in 4, 100, and 400 year cycles. That these have * the correct values is asserted in the module init function. */ #define DI4Y 1461 /* days_before_year(5); days in 4 years */ #define DI100Y 36524 /* days_before_year(101); days in 100 years */ #define DI400Y 146097 /* days_before_year(401); days in 400 years */ ... skipping to the init function ... /* A 4-year cycle has an extra leap day over what we'd get from * pasting together 4 single years. */ assert(DI4Y == 4 * 365 + 1); assert(DI4Y == days_before_year(4+1)); This is probably explainable by the limitations of the C language, but I would find _DI4Y = 4 * 365 + 1 assert _DI4Y == _days_before_year(5) to be more natural than the way it is currently written.

Ain't broke - don't fix.

Agree.

On Sat, Nov 16, 2013 at 5:09 PM, Nick Coghlan wrote:

On 16 November 2013 23:17, Maciej Fijalkowski wrote:

...

On Sat, Nov 16, 2013 at 3:51 AM, Terry Reedy wrote:

...

If user input can trigger an assert, then the code should raise a normal exception that will not disappear with -OO.

May I assert that -OO should instead be killed?

"I don't care about embedded devices" is not a good rationale for killing features that really only benefit people running Python on such systems.

Cheers, Nick.

Can I see some writeup how -OO benefit embedded devices?

On Sat, 16 Nov 2013 17:34:15 +0200 Maciej Fijalkowski wrote:

On Sat, Nov 16, 2013 at 5:33 PM, Maciej Fijalkowski wrote:

...

On Sat, Nov 16, 2013 at 5:09 PM, Nick Coghlan wrote:

...

On 16 November 2013 23:17, Maciej Fijalkowski wrote:

...

On Sat, Nov 16, 2013 at 3:51 AM, Terry Reedy wrote:

...

If user input can trigger an assert, then the code should raise a normal exception that will not disappear with -OO.

May I assert that -OO should instead be killed?

"I don't care about embedded devices" is not a good rationale for killing features that really only benefit people running Python on such systems.

Cheers, Nick.

Can I see some writeup how -OO benefit embedded devices?

Or more importantly, how removing assert does. And how not naming it --remove-asserts would not help (people really have an opinion it would optimize their code)

I agree that conflating the two doesn't help the discussion. While removing docstrings may be beneficial on memory-constrained devices, I can't remember a single situation where I've wanted to remove asserts on a production system. (I also tend to write less and less asserts in production code, since all of them tend to go in unit tests instead, with the help of e.g. mock objects) Regards Antoine.

Personally I think that none of the -O* should be removing asserts. It feels like a foot gun to me. I’ve seen more than one codebase that would be completely broken under -O* because they used asserts without even knowing -O* existed. Removing __debug__ blogs and doc strings I don’t think is as big of a deal, although removing doc strings can break code as well. On Nov 16, 2013, at 11:08 AM, Nick Coghlan wrote:

On 17 November 2013 01:46, Antoine Pitrou wrote:

...

I agree that conflating the two doesn't help the discussion. While removing docstrings may be beneficial on memory-constrained devices, I can't remember a single situation where I've wanted to remove asserts on a production system.

While I actually agree that having separate flags for --omit-debug, --omit-asserts and --omit-docstrings would make more sense than the current optimization levels, Maciej first proposed killing off -OO (where the most significant effect is removing docstrings which can result in substantial program footprint reductions for embedded systems), and only later switched to asking about removing asserts (part of -O, which also removes blocks guarded by "if __debug__", both of which help embedded systems preserve precious ROM space, although to a lesser degree than removing docstrings can save RAM).

One of the most important questions to ask when proposing the removal of something is "What replacement are we offering for those users that actually need (or even just think they need) this feature?". Sometimes the answer is "Nothing", sometimes it's something that only covers a subset of previous use cases, and sometimes it's a complete functional equivalent with an improved spelling. But not asking the question at all (or, worse, dismissing the concerns of affected users as irrelevant and uninteresting) is a guaranteed way to annoy the very people that actually rely on the feature that is up for removal or replacement, when you *really* want them engaged and clearly explaining their use cases.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Hi, Some languages (C#, java) do the reverse by removing assertions if we don't tell compiler to keep them. Personnaly, I find this solution relatively accurate as I expect assertions not to be run in production. It would be painful to have this behaviour in python now, but I hope we'll keep a way to remove assertions and find interesting the solution of specific flags (--omit-debug, --omit-asserts and --omit-docstrings). cheers, Grégory 2013/11/16 Donald Stufft

Personally I think that none of the -O* should be removing asserts. It feels like a foot gun to me. I’ve seen more than one codebase that would be completely broken under -O* because they used asserts without even knowing -O* existed.

Removing __debug__ blogs and doc strings I don’t think is as big of a deal, although removing doc strings can break code as well.

On Nov 16, 2013, at 11:08 AM, Nick Coghlan wrote:

...

On 17 November 2013 01:46, Antoine Pitrou wrote:

...

I agree that conflating the two doesn't help the discussion. While removing docstrings may be beneficial on memory-constrained devices, I can't remember a single situation where I've wanted to remove asserts on a production system.

While I actually agree that having separate flags for --omit-debug, --omit-asserts and --omit-docstrings would make more sense than the current optimization levels, Maciej first proposed killing off -OO (where the most significant effect is removing docstrings which can result in substantial program footprint reductions for embedded systems), and only later switched to asking about removing asserts (part of -O, which also removes blocks guarded by "if __debug__", both of which help embedded systems preserve precious ROM space, although to a lesser degree than removing docstrings can save RAM).

One of the most important questions to ask when proposing the removal of something is "What replacement are we offering for those users that actually need (or even just think they need) this feature?". Sometimes the answer is "Nothing", sometimes it's something that only covers a subset of previous use cases, and sometimes it's a complete functional equivalent with an improved spelling. But not asking the question at all (or, worse, dismissing the concerns of affected users as irrelevant and uninteresting) is a guaranteed way to annoy the very people that actually rely on the feature that is up for removal or replacement, when you *really* want them engaged and clearly explaining their use cases.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/apieum%40gmail.com

On Saturday, November 16, 2013, Steven D'Aprano wrote:

On Sat, Nov 16, 2013 at 11:16:48AM -0500, Donald Stufft wrote:

...

Personally I think that none of the -O* should be removing asserts. It feels like a foot gun to me. I’ve seen more than one codebase that would be completely broken under -O* because they used asserts without even knowing -O* existed.

I agree that many people misuse asserts. I agree that this is a problem for them. But I disagree that the solution is to remove what I consider to be a useful feature, and a common one in other languages, just to protect people from their broken code.

I prefer to see better efforts to educate them. To that end, I've taken the liberty of sending a post to python-list@python.org javascript:;describing when to use asserts, and when not to use them:

https://mail.python.org/pipermail/python-list/2013-November/660401.html

Nice post!

I will continue to use my best effort to encourage good use of assertions in Python.

Thank you! --Guido

-- --Guido van Rossum (on iPad)

On Sun, 17 Nov 2013 18:04:43 +1100 Steven D'Aprano wrote:

On Sat, Nov 16, 2013 at 04:46:00PM +0100, Antoine Pitrou wrote:

...

I agree that conflating the two doesn't help the discussion. While removing docstrings may be beneficial on memory-constrained devices, I can't remember a single situation where I've wanted to remove asserts on a production system.

(I also tend to write less and less asserts in production code, since all of them tend to go in unit tests instead, with the help of e.g. mock objects)

I'm the opposite. I like using asserts in my code, and while I don't *typically* run it with -O, I do think it is valuable to have to opportunity to remove asserts. I've certainly written code where -O has given a major micro-optimization of individual functions (anything up to 50% speedup). I've never measured if that lead to a significant whole-application speedup, but I assume that was some benefit. (I know, premature optimization and all that...)

So you assumed there was some benefit, but you never used it anyway? Regards Antoine.

On Sun, 17 Nov 2013 21:27:24 +1100 Steven D'Aprano wrote:

On Sun, Nov 17, 2013 at 11:00:50AM +0100, Antoine Pitrou wrote:

...

On Sun, 17 Nov 2013 18:04:43 +1100 Steven D'Aprano wrote:

...

On Sat, Nov 16, 2013 at 04:46:00PM +0100, Antoine Pitrou wrote:

...

I agree that conflating the two doesn't help the discussion. While removing docstrings may be beneficial on memory-constrained devices, I can't remember a single situation where I've wanted to remove asserts on a production system.

(I also tend to write less and less asserts in production code, since all of them tend to go in unit tests instead, with the help of e.g. mock objects)

I'm the opposite. I like using asserts in my code, and while I don't *typically* run it with -O, I do think it is valuable to have to opportunity to remove asserts. I've certainly written code where -O has given a major micro-optimization of individual functions (anything up to 50% speedup). I've never measured if that lead to a significant whole-application speedup, but I assume that was some benefit. (I know, premature optimization and all that...)

So you assumed there was some benefit, but you never used it anyway?

I had a demonstrable non-trivial speedup when timing individual functions. At the time I considered that "good enough". If you want to call that "premature optimization", I can't entirely disagree, but can you honestly say you've never done the same?

You didn't answer my question: did you actually use -OO in production, or not? Saying that -OO could have helped you optimize something you didn't care about isn't a very strong argument for -OO :) What I would like to know is if people *knowingly* add costly asserts to performance-critical code, with the intent of disabling them at runtime using -OO. (in Python, that is, I know people do that in C where the programming culture and tools are different) Regards Antoine.

Le 17/11/2013 12:27, Steven D'Aprano a écrit :

...

What I would like to know is if people *knowingly* add costly asserts to performance-critical code, with the intent of disabling them at runtime using -OO. Yes, I have knowingly added costly asserts to code with the intend of disabling them at runtime. Was it *performance-critical* code? I don't know, that was the point of my earlier rambling -- I could demonstrate a speedup of the individual functions in benchmarks, but nobody spent the effort to determine which functions were performance critical.

Hi, my 2 cents: asserts have been of a great help in the robustness of our provisioning framework, these are like tests embedded in code, to *consistently* check what would be VERY hard to test from the outside, from unit-tests. It makes us gain much time when we develop, because asserts (often used for "method contract checking") immediately break stuffs if we make dumb programming errors, like giving the wrong type of variable as parameter etc. (if you send a string instead of a list of strings to a method, it could take a while before the errors gets noticed, since their behaviour is quite close) We also add asserts with very expensive operations (like fully checking the proper synchronization of our DBs with the mockups of remote partners, after each provisioning command treated), so that we don't need to call something like that after every line of unit-test we write. In production, we then make sure we use -O flag to avoid doubling our treatments times and traffic. regards, Pascal

Gregory Salvan wrote:

I believe the point of removing assertions is also to avoid throwing unhandled developper errors to end-user

I'm not sure I buy that. An assert is saying that something should never happen. If it does happen, either it's going to lead to an exception further down the track, or produce incorrect results. In the first case, you've failed to shield the user from exceptions; in the second, better to log an exception than silently pretend nothing has gone wrong. -- Greg

On Sun, Nov 17, 2013 at 01:50:31AM +1000, Nick Coghlan wrote:

No, that's the wrong question to ask. The onus is on *you* to ask "Who is this feature for? Do they still need it? Can we meet their needs in a different way?". You're the one proposing to break things, so it's up to you to make the case for why that's an OK thing to do. [...] That's not the way this works - backwards compatibility is sacrosanct, and it requires some seriously compelling evidence to justify a breach.

Thanks for saying this. I get frustrated by the number of times people propose removing -O (apparently) just because they personally don't use it. I personally have never got any benefit from the tarfile or multiprocessing modules, but I don't ask for them to be removed :-) -- Steven

On Sun, 17 Nov 2013 17:14:38 +0100, Victor Stinner wrote:

2013/11/16 Maciej Fijalkowski :

...

Can I see some writeup how -OO benefit embedded devices?

You get smaller .pyc files. In an embedded device, the whole OS may be written in a small memory, something like 64 MB or smaller. Removing doctrings help to fit in 64 MB.

I don't know if dropping "assert" statements is required on embedded device.

I've worked on a project (mobile platform) where both of these were true. Did we gain much by dropping asserts? As with Steve's case, no one bothered to measure, but it was a no-brainer win so we did it. Especially since we really needed to kill the docstrings to save memory. (Killing linecache was an even bigger win on the memory side). I unfortunately don't remember how much difference removing docstrings made in the process size, but I'm pretty sure it was more than a MB, and on a device with no swap in a process that must be running all the time, even 1MB makes a significant difference. On the assert side, if there hadn't been an option to remove the asserts, I'm sure that the call would have come down to manually remove all the asserts for production, which would have been a pain. In that case they *might* have measured, but since you'd have to do the removal work to do the measurement, they'd probably not have bothered. I'd say that if someone wants to drop assert removal, they'd need to prove that it *didn't* result in a speedup, which would be pretty hard considering that the checks covered by assert/if __debug__ can be arbitrarily complex :) But...yes, it would have been nice to have been able to remove docstrings and asserts *separately*. We might have measured the delta and decided to keep them in the beta in that case, since the asserts did catch some bugs. --David

On Sun, Nov 17, 2013 at 9:02 PM, Barry Warsaw wrote:

On Nov 17, 2013, at 05:14 PM, Victor Stinner wrote:

...

2013/11/16 Maciej Fijalkowski :

...

Can I see some writeup how -OO benefit embedded devices?

You get smaller .pyc files. In an embedded device, the whole OS may be written in a small memory, something like 64 MB or smaller. Removing doctrings help to fit in 64 MB.

I'm in support of separate flags for stripping docstrings and asserts. I'd even be fine with eliminating a flag to strip docstrings if we had a post-processing tool that you could apply to pyc files to strip out the docstrings. Another problem that I had while addressing these options in Debian was the use of .pyo for both -O and -OO level.

-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

My problem with -O and -OO is that their arguments are very circular. Indeed, I understand the need why you would want in certain and limited cases to remove both docstrings and asserts. So some options for doing so are ok. But a lot of arguments I see are along the lines of "don't use asserts because -O removes them". If the option was named --remove-asserts, noone would care, but people care since -O is documented as "do optimizations" and people *assume* this is what it does (makes code faster) and as unintended consequence removes asserts. Cheers, fijal

17.11.2013 23:05, Guido van Rossum wrote:

The correct rule should be "don't use assert (the statement) to check for valid user input" and the stated reason should be that the assert statement was *designed* to be disabled globally, not to be a shorthand for "if not X: raise (mumble) Y". A corollary should also be that unittests should not use the assert statement; some frameworks sadly encourage the anti-pattern of using it in tests.

My problem with -O (and -OO) is that even though my code is valid (in terms of the rule 'use assert only for should-never-happen cases') I have no control over 3rd party library code: I can never know whether doesn't it break if I turn -O or -OO on (as long as I do not analyze carefully the code of the libraries I use, including writing regression tests [for 3rd party code]...). Woudln't it to be useful to add possibility to place an "optimisation cookie" (syntactically analogous to "encoding cookie") at the beginning of each of my source files (when I know they are "-O"-safe), e.g.: # -*- opt: asserts -*- or even combined with an encoding cookie: # -*- coding: utf-8; opt: asserts, docstrings -*- Then: * The -O flag would be effectively applied *only* to a file containing such a cookie and *exactly* according to the cookie content (whether asserts, whether docstrings...). * Running without -O/-OO would mean ignoring optimisation cookies. * The -OO flag would mean removing both asserts and docstrings (i.e. the status quo of -OO). * Fine-grained explicit command line flags such as --remove-asserts and --remove-docstings could also be useful. (Of course, the '-*-' fragments in the above examples are purely conventional; the actual regex would not include them as it does not include them now for encoding cookies.) Cheers. *j