GNU bug report logs - #72524
how does grep determine locale if no LC environment variables are set

Previous Next

Full log

Message #11 received at 72524 <at> debbugs.gnu.org (full text, mbox):

From: <mark.yagnatinsky <at> barclays.com>
To: <eggert <at> cs.ucla.edu>
Cc: 72524 <at> debbugs.gnu.org
Subject: RE: bug#72524: how does grep determine locale if no LC environment
 variables are set
Date: Thu, 8 Aug 2024 19:24:51 +0000

Re: how am I doing that ... via bash, just like the way you suggested I run "locale" the second time:
LC_CTYPE=C.UTF-8 grep -P needle haystack.txt  # just CTYPE seems to be enough, no need for ALL

Re: output difference for locale:
First way, everything is C.UTF-8 except LANG and LC_ALL, which are blank
Second way: same thing, except LC_ALL is no longer blank

Re: is_using_utf8 ... It relies on mbrtowc, which in turn relies on the current locale.
It seems that this function should NEVER return false in a UTF-8 locale.
But how does grep decide what the locale even is?
Presumably it must call setlocale at some point, or else it would be using the C locale, which is surely a unibyte locale.
Does it just do the obvious thing and call it with the empty string?
If so, is there any good way to find out what locale it actually got "resolved" to?

-----Original Message-----
From: Paul Eggert <eggert <at> cs.ucla.edu> 
Sent: Thursday, August 8, 2024 12:10 PM
To: Yagnatinsky, Mark : IT (NYK) <mark.yagnatinsky <at> barclays.com>
Cc: 72524 <at> debbugs.gnu.org
Subject: Re: bug#72524: how does grep determine locale if no LC environment variables are set

 CAUTION:  This email originated from outside our organization - eggert <at> cs.ucla.edu  Do not click on links, open attachments, or respond unless you recognize the sender and can validate the content is safe. 

______________________________________________________________________
On 2024-08-08 05:53, mark.yagnatinsky--- via Bug reports for GNU grep wrote:
> I ran into an odd issue... the workaround is easy enough but the issue is weird.
> In case this relevant, my grep coms from git bash.  (which I think is 
> mostly Cygwin? (or maybe msys2??)) Anyway, grep -P doesn't work if no LC vars are set, and complains that it only works in unibyte locales or UTF-8.
> Normally, the git bash mintty launcher sets LC_CTYPE to en_us.UTF-8 but not if I bypass the launcher and run grep directly.
> Here's the weird part, if I ask /usr/bin/locale what LC_TYPE "should" be, it says C.UTF-8.
> If I run grep with C.UTF-8 then it also works.  So it must be deriving a default locale an different way.

My guess is that your default environment says that it supports UTF-8, but it doesn't support it well enough to pass grep's test; see grep/lib/localeinfo.c's is_using_utf8. If my guess is right, you may be encountering subtle bugs in programs other than grep.

When you say "I run grep with C.UTF-8" how exactly do you do that?

Is there any difference in output between these two shell commands?

localeinfo
LC_ALL=C.UTF-8 localeinfo

If you have a debugger, you might look into why is_using_utf8 returns false in your default locale.

This message is for information purposes only. It is not a recommendation, advice, offer or solicitation to buy or sell a product or service, nor an official confirmation of any transaction. It is directed at persons who are professionals and is intended for the recipient(s) only. It is not directed at retail customers. This message is subject to the terms at: https://www.ib.barclays/disclosures/web-and-email-disclaimer.html. 

For important disclosures, please see: https://www.ib.barclays/disclosures/sales-and-trading-disclaimer.html regarding marketing commentary from Barclays Sales and/or Trading desks, who are active market participants; https://www.ib.barclays/disclosures/barclays-global-markets-disclosures.html regarding our standard terms for Barclays Investment Bank where we trade with you in principal-to-principal wholesale markets transactions; and in respect to Barclays Research, including disclosures relating to specific issuers, see: https://publicresearch.barclays.com.
__________________________________________________________________________________ 
If you are incorporated or operating in Australia, read these important disclosures: https://www.ib.barclays/disclosures/important-disclosures-asia-pacific.html.
__________________________________________________________________________________
For more details about how we use personal information, see our privacy notice: https://www.ib.barclays/disclosures/personal-information-use.html. 
__________________________________________________________________________________

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.