GNU bug report logs - #60690
[PATCH v2] grep: correctly identify utf-8 characters with \{b,w} in -P

Previous Next

Full log

View this message in rfc822 format

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: demerphq <at> gmail.com, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01 <at> gmail.com>, Carlo Marcelo Arenas Belón <carenas <at> gmail.com>, Ævar Arnfjörð Bjarmason <avarab <at> gmail.com>, gitster <at> pobox.com, pcre-dev <at> exim.org, Tukusej’s Sirs <tukusejssirs <at> protonmail.com>, git <at> vger.kernel.org
Subject: bug#60690: -P '\d' in GNU and git grep
Date: Tue, 4 Apr 2023 08:31:44 -0700

On Mon, Apr 3, 2023 at 11:47 PM Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> On 2023-04-03 20:30, Jim Meyering wrote:
> > have you seen justification
> > (other than for compatibility with some other tool or language) for
> > allowing \d to match non-ASCII by default, in spite of the risks?
>
> In the example Ævar supplied in <https://bugs.gnu.org/60690>, my
> impression was that it was better when \d matched non-ASCII digits. That
> is, in a UTF-8 locale it's better when \d finds matches in these lines:
>
> >>      > git-gui/po/ja.po:"- 第１行: 何をしたか、を１行で要約。\n"
> >>      > git-gui/po/ja.po:"- 第２行: 空白\n"
>
> because they contain the Japanese digits "１" and "２". This was the only
> example I recall being given.

Before it was unintentionally enabled in grep-3.9, lines like that have
never been matched by grep -P's '\d'. By relaxing \d, we'd weaken
any application that uses say grep -P '^\d+$' to perform input
validation intending to ensure that some input is all ASCII digits.
It's not a big stretch to imagine that some downstream processor
of that "verified" data is not prepared to deal with multi-byte digits.

> Also, I find it odd that grep -P '^[\w\d]*$' matches lines containing
> any sort of Arabic word characters, but it rejects lines containing
> Arabic digits like "٣" that are perfectly reasonable in Arabic-language
> text. I also find it odd that [\d] and [[:digit:]] mean different things.
>
> There are arguments on the other side, otherwise we wouldn't be having
> this discussion. And it's true that grep -P '\d' formerly rejected
> Arabic digits (though it's also true that grep -P '\w' formerly rejected
> Arabic letters...). Still, the cure's oddness and incompatibility with
> Git, Perl, etc. appears to me to be worse than the disease of dealing
> with grep -P invocations that need to use [0-9] or LC_ALL="C" anyway if
> they want to be portable to any program other than GNU grep.

I'm primarily concerned about not introducing a persistent regression in
how GNU grep's -P '\d' works in multibyte locales. The corner cases you
mention do matter, of course, but are far less likely to matter in practice.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.