It'd be good to know just how much benefit this precompilation actually grants.

As far as I know, Pattern objects in regex module can be pickled, don't know if it's useful.

...

...

import pickle import regex p = regex.compile('[a-z]') b = pickle.dumps(p) p = pickle.loads(b)

Wow that's an old post of mine I searched on Google before post this, hope there is no omission.

Reply to Stefan Behnel and Chris Angelico. On 18-12-27 22:42, Stefan Behnel wrote:

  >>> import pickle, re   >>> p = re.compile("[abc]")   >>> pickle.dumps(p)   b'\x80\x03cre\n_compile\nq\x00X\x05\x00\x00\x00[abc]q\x01K \x86q\x02Rq\x03.'

What this does, essentially, is to make the pickle loader pass the original regex pattern string into re.compile() to "unpickle" it. Meaning, it compiles the regex on the way in. Thus, there isn't much to gain from using (the current form of) regex pickling here.

Yes, re module only pickles pattern string and flags, it's safe for cross-version pickle/unpickle. re module's pickle code: def _pickle(p):     return _compile, (p.pattern, p.flags) copyreg.pickle(Pattern, _pickle, _compile) On 18-12-28 1:27, Chris Angelico wrote:

What Stefan pointed out regarding the stdlib's "re" module is also true of the third party "regex" - unpickling just compiles from the original string.

I had followed regex module for a year, it does pickle the compiled data, this is its code: def _pickle(pattern):     return _regex.compile, pattern._pickled_data _copy_reg.pickle(Pattern, _pickle) // in _regex.c file self->pickled_data = Py_BuildValue("OnOOOOOnOnn", pattern, flags,     code_list, groupindex, indexgroup, named_lists, named_list_indexes,     req_offset, required_chars, req_flags, public_group_count); if (!self->pickled_data) {     Py_DECREF(self);     return NULL; }

On Thu, Dec 27, 2018 at 05:47:46PM +0000, MRAB wrote:

On 2018-12-27 11:48, Ma Lin wrote: [snip]

...

2, We can't use regex module as a drop-in replacement: import regex as re IMHO, I would like to see regex module be adopted into stdlib after cutting off its "full case-folding" and "fuzzy matching" features.

I think that omitting full casefolding would be a bad idea; after all, strings (in Python 3) have a .casefold method.

And I don't understand why omitting fuzzy matching is a good idea. If you don't want fuzzy matching, don't use it in your code. But why remove it? -- Steve

On Fri, Dec 28, 2018 at 1:56 AM Ma Lin wrote:

Maybe this literal will encourage people to finish tasks using regex, even lead to abuse regex, will this change Python's style?

What's worse is, people using mixed manners in the same project:

one_line.split(',') ... p','.split(one_line)

Maybe it will break the Python's style, reduce code readability, is this worth it?

The bar for introducing a new type of literal should be very high. Do performance numbers show this change would have a large impact for a large amount of libraries and programs? In my opinion, only if this change would make 50% of programs run 50% faster then it might be worth discussing. The damage to readability, burden of changing syntax and burden of yet another language feature for newcomers to learn is too high. Cheers, Yuval

regex = re.compile(r"...") regex = p("...")

is not that much different.

True, but when the literal is put somewhere far from the compile() call it becomes a problem for static analysis. Conceptually a regex is not a string but an embedded foreign language. That's why I think this discussion is worth having. It would be nice with a way to mark up foreign languages in a way that had some other advantages so people would be incentivised to do it, but just a way to mark it with comments would be fine too I think if it's standardized. Maybe the discussion should be expanded to cover the general case of embedded foreign languages? SQL, HTML, CSS and (obviously) regex comes to mind. One could also think of C for stuff like CFFI. / Anders

On 18-12-28 22:54, Joao S. O. Bueno wrote:

Sorry for sounding over-reactive, but yes, this could make Python look like Perl. Yes, this may introduce Perl's style irreversibly, we need to be cautious about this.

I'm thinking, if people ask these questions in their mind when reading a piece of Python code: 1, "Is this Python code?" 2, "What's the purpose of this code?" 3, "How can I modify it if I want to ... ?" Maybe Python is on a doubtful way. There is an interesting question: Will literal p"" ruin Python's (or other dynamic languages like Ruby) style? Why will this happen?

And a 3rd shortcoming - flags can't be passed as parameters, and have to be built-in the regexp themselves, further complicating the readability even for very simple regular expressions. IMO this is an advantage, it's hard to omit flags when reading/copying an regex pattern.

On Sat, Dec 29, 2018 at 04:29:32PM +1100, Alexander Heger wrote:

for regular strings one can write

"aaa" + "bbb"

which also works for f-strings, r-strings, etc.; in regular expressions, there is, e.g., parameter counting and references to numbered matches. How would that be dealt with in a compound p-string? Either it would have to re-compiled or not, either way could lead to unexpected results

What does Perl do?

p"(\d)\1" + p"(\s)\1"

Since + is used for concatenation, then that would obviously be the same as: p"(\d)\1(\s)\1" Whether it gets done at compile-time or run-time depends on how smart the keyhole optimiser is. If it is smart enough to recognise regex literals, it could fold the two strings together and regex-compile them at python-compile time, otherwise it could be equivalent to: _t1 = re.compile(r"(\d)\1") # compile-time _t2 = re.compile(r"(\s)\1") # compile-time re.compile(_t1.pattern + _t2.pattern) # run-time Obviously that defeats the purpose of using a p"" pre-compiled regex object, but the answer to that is either: 1. Don't do that then; or 2. We better make sure the keyhole optimizer is smarter. Or we just ban concatenation. "P-strings" aren't strings, even though they look like them.

This brings me to the point that the key difference is that f- and r- strings actually return strings,

To be precise, f-"strings" are actually code that returns a string when executed at runtime; r-strings are literal syntax for strings.

whereas p- string would return a different kind of object. That would seem certainly very confusing to novices - and also for the language standard as a whole.

Indeed. Perhaps something like \\regex\\ would be better, *if* this feature is desired. -- Steve

On Sat, Dec 29, 2018 at 12:30 AM Alexander Heger wrote:

for regular strings one can write

"aaa" + "bbb"

which also works for f-strings, r-strings, etc.; in regular expressions, there is, e.g., parameter counting and references to numbered matches. How would that be dealt with in a compound p-string? Either it would have to re-compiled or not, either way could lead to unexpected results

p"(\d)\1" + p"(\s)\1"

or

p"^(\w)" + p"^(\d)"

regular strings can be added, bu the results of p-string could not - well, their are not strings.

Isn't this a feature, not a bug, of encouraging literals to be specified as patterns: addition of patterns would raise an error (as is currently the case for addition of compiled patterns in the re and regex modules)? Currently, I find it easiest to use r-strings for patterns and call re.search() etc. without precompiling them, which means that I could accidentally concatenate two patterns together that would silently produce an unmatchable pattern. Using p-literals for most patterns would mean I have to be explicit in the exceptional case where I do want to assemble a pattern from multiple parts: FIRSTNAME = p"[A-Z][-A-Za-z']+" LASTNAME = p"[-A-Za-z']([-A-Za-z' ]+[-A-Za-z'])?" FULLNAME = FIRSTNAME + p' ' + LASTNAME # error FIRSTNAME = r"[A-Z][-A-Za-z']+" LASTNAME = r"[-A-Za-z']([-A-Za-z' ]+[-A-Za-z'])?" FULLNAME = re.compile(FIRSTNAME + ' ' + LASTNAME) # success Another potential advantage is that an ill-formed p-literal (such as a mismatched parenthesis) would be caught immediately, rather than when it is first used. This could pay off, for example, if I am defining a data structure with a bunch of regexes that would get used for different input. (But there may be performance tradeoffs here.)

This brings me to the point that the key difference is that f- and r- strings actually return strings, whereas p- string would return a different kind of object. That would seem certainly very confusing to novices - and also for the language standard as a whole.

The b prefix produces a bytes literal. Is a bytes object a kind of string, more so than a regex pattern is? I could see an argument that bytes is a particular encoding of sequential character data, whereas a regex pattern represents a string *language*, i.e. an abstraction over string data. But...this distinction starts to feel very theoretical rather than practical. If novices are expected to read code with regular expressions in it, why would they have trouble understanding that the "p" prefix means "pattern"? As someone who works with text a lot, I think there's a decent practicality-beats-purity argument in favor of p-literals, which would make regex operations more easily accessible and prevent patterns from being mixed up with string data. A potential downside, though, is that it will be tempting to introduce flags as prefixes, too. Do we want to go down the road of pui"my Unicode-compatible case-insensitive pattern"? Nathan

-Alexander

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

What's more, it's a tool that should be used with considerable reluctance, because REs are essentially unreadable, so every time you use one you're creating a maintenance headache.

Well, it requires some experience to read REs, I have written many, and I still need to test thoroughly even many basic ones for that they really do what they are supposed to do. And then there is the issue that there is many different implementation, what you have to escape, etc., varies between python (raw and regular strings), emacs, grep, overleaf, ... Never mind, my main point is that they return an object that is qualitatively different from a string, for example, in terms of concatenation. I also think it is too specialised, and time-critical constant REs can be stored in the module body, etc., if need be. I do that. But since this is the ideas mailing list, and taking this thread on an excursion, maybe an "addition" operator could be defined for REs, such that re.compile(s1 + s1) == re.compile(s1) + re.compile(s2) with the restriction that s1 and s2 are strings that are valid REs each. Even that would leave questions about how to deal with compile flags; they probably should be treated the same as if they were embedded at the beginning of each string. -Alexander

Le 31/12/2018 à 12:31, M.-A. Lemburg a écrit :

We already have re.search() and re.match() which deal with compilation on-the-fly and caching. Perhaps the documentation should hint at this more explicitly...

The complaint is that the global cache is still too costly. See measurements in https://bugs.python.org/issue35559 Regards Antoine.

Ma Lin schrieb am 31.12.18 um 14:02:

On 18-12-31 19:47, Antoine Pitrou wrote:

...

The complaint is that the global cache is still too costly. See measurements in https://bugs.python.org/issue35559

In this issue, using a global variable `_has_non_base16_digits` [1] will accelerate 30%. Is re module's internal cache [2] so bad?

If rewrite re module's cache with C and use a custom data structure, maybe we will get a small speedup.

[1] `_has_non_base16_digits` in PR11287 [1] https://github.com/python/cpython/pull/11287/files

[2] re module's internal cache code: [2] https://github.com/python/cpython/blob/master/Lib/re.py#L268-L295

_cache = {}  # ordered! _MAXCACHE = 512 def _compile(pattern, flags):     # internal: compile pattern     if isinstance(flags, RegexFlag):         flags = flags.value     try:         return _cache[type(pattern), pattern, flags]     except KeyError:         pass     ...

I wouldn't be surprised if the slowest part here was the isinstance() check. Maybe the RegexFlag class could implement "__hash__()" as "return hash(self.value)" ? Stefan