GNU bug report logs - #60690
[PATCH v2] grep: correctly identify utf-8 characters with \{b,w} in -P

Previous Next

Full log

Message #40 received at 60690 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Junio C Hamano <gitster <at> pobox.com>
Cc: demerphq <at> gmail.com, Philip.Hazel <at> gmail.com, 60690 <at> debbugs.gnu.org,
 mega lith01 <megalith01 <at> gmail.com>, Carlo Arenas <carenas <at> gmail.com>,
 Ævar Arnfjörð Bjarmason <avarab <at> gmail.com>,
 pcre-dev <at> exim.org,
 Tukusej’s Sirs <tukusejssirs <at> protonmail.com>,
 git <at> vger.kernel.org
Subject: Re: bug#60690: -P '\d' in GNU and git grep
Date: Wed, 5 Apr 2023 11:32:38 -0700

On 2023-04-04 12:31, Junio C Hamano wrote:

> My personal inclination is to let Perl folks decide
> and follow them (even though I am skeptical about the wisdom of
> letting '\d' match anything other than [0-9])

I looked into what pcre2grep does. It has always done only 8-bit 
processing unless you use the -u or --utf option, so plain "pcre2grep 
'\d'" matches only ASCII digits.

Although this causes pcre2grep to mishandle Unicode characters:

  $ echo 'Ævar' | pcre2grep '[Ssß]'
  Ævar

it mimics Perl 5.36:

  $ echo 'Ævar' | perl -ne 'print $_ if /[Ssß]/'
  Ævar

so this seems to be what Perl users expect, despite its infelicities.

For better Unicode handling one can use pcre2grep's -u or --utf option, 
which causes pcre2grep to behave more like GNU grep -P and git grep -P: 
"echo 'Ævar' | pcre2grep -u '[Ssß]'" outputs nothing, which I think is 
what most people would expect (unless they're Perl users :-).

Neither git grep -P nor the current release of pcre2grep -u have \d 
matching non-ASCII digits, because they do not use PCRE2_UCP. However, 
in a February 8 commit[1], Philip Hazel changed pcre2grep to use 
PCRE2_UCP, so this will mean 10.43 pcre2grep -u will behave like 3.9 GNU 
grep -P did (though 3.10 has changed this).

That February commit also added a --no-ucp option, to disable PCRE2_UCP. 
So as I understand it, if you're in a UTF-8 locale:

* 10.43 pcre2grep -u will behave like 3.9 GNU grep -P.

* 10.43 pcre2grep -u --no-ucp will behave like git grep -P.

* Current GNU grep -P is different from everybody else.

This incompatibility is not good.

Here are two ways forward to fix this incompatibility (there are other 
possibilities of course):

(A) GNU grep adds a --no-ucp option that acts like 10.43 pcre2grep 
--no-ucp, and git grep -P follows suit. That is, both GNU and git grep 
act like 10.43 pcre2grep -u, in that they enable PCRE2_UTF, and also 
enable PCRE2_UCP unless --no-ucp is given. This would cause \d to match 
non-ASCII digits unless --no-ucp is given.

(B) GNU grep -P and git grep -P mimic pcre2grep in both -u and --no-ucp. 
That is, they would both do 8-bit-only by default, and use PCRE2_UTF 
only when -u or --utf is given, and use PCRE2_UCP only when --no-ucp is 
absent. This would cause \d to match non-ASCII digits only when -u is 
given but --no-ucp is not.

Under either (A) or (B), future pcre2grep -u, GNU grep -P, and git grep 
-P would be consistent.

I mildly prefer (B) but (A) would also work. (One advantage of (B) is 
that it should be faster....)

[1]: 
https://github.com/PCRE2Project/pcre2/commit/8385df8c97b6f8069a48e600c7e4e94cc3e3ebd9ht

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.